Freesurfer Test Plan
WORK IN PROGRESS
Introduction
This page documents the Freesurfer software test plan. A formal test plan ([http://en.wikipedia.org/wiki/Test_plan see Wikipedia reference]) gives detailed testing information regarding an upcoming testing effort, including:
- Scope of testing
- Schedule
- Test deliverables
- Release criteria
- Risks and contingencies
Tests should cover the following categories of testing:
- Functional - can the software executes its basic functionality under optimal conditions?
- Boundary - determine the breaking points of the software, and whether the software gracefully handles input near and beyond these boundaries.
- Stability - gauge long-term behavior of the software: whether is has a memory leak, or prone to crashes which are not repeatable in any single run of any of the other tests.
- Coverage - what percentage of the code-base is exercised by the tests?
- Performance - produce benchmarks on the performance of the software.
The Freesurfer test plan is a work-in-progress. It is one not developed top-down, but rather grown from the bottom-up as necessity and time dictate. The goal is to build a test suite that meets the criteria of a formal test plan. This will take time.
The current test suite is an ad-hoc collection of test scripts providing rudimentary testing of most of the freesurfer code-base consisting of unit, module and system tests.
Unit tests
The term 'unit test' is defined in our Freesurfer test plan to mean a test of a freesurfer binary (such as mri_convert) or smaller (a subroutine). The framework for these tests is the 'make check' framework built into the 'make' utility (and the 'automake' tools). The 'check' target of 'make' initiates the build and run of tests created by the user to test the thing that is made by the 'all' target. In freesurfer, there are a number of 'make check' tests, and 'make check' is run after 'make' on each nightly build platform (see the section [wiki:DevelopersGuide/MartinosCenter "How the nightly build works"] for details).
Module tests
- aseg tests
- aparc tests
System tests
test_recon-all - see the section [wiki:DevelopersGuide/MartinosCenter "How the daily testing works"] for details)
Reporting
- Dart