## page was renamed from Buckner40Testing
== Buckner40 and Adni60 Testing ==
''see also: TestPlan, and Buckner40Notes''

=== Introduction ===
The 'Buckner40' is a name given to a small subset of subjects originating from Washington University, collected
by Randy Buckner and colleagues. It consists of a group of 40 nondemented subjects, ranging in age from 18 to 93
(with no subjects between 30 to 65 years of age), 21 women and 19 men. These data are a subset of a larger open-access structural data set created by the Buckner lab.  That data may (or may not) be available at the time of this writing, but you can try the link https://cnl.rc.fas.harvard.edu/data-tools. Previously, it was referenced by  accession number 2-2004-1168X.

For Freesurfer, this dataset is especially nice as a test-set because of the wide subject age range, and the quality of the scans, which, while good, are not great, but are typical of what Freesurfer has to handle.

The Buckner40Notes wiki page documents the work of examining these brains, slice by slice, to verify their correctness, in terms of how well Freesurfer has reconstructed the white and pial surfaces, and segmented the subcortical structures.

It is important that Freesurfer's recon-all processing stream continue to process all of the subjects successfully, where that means the results are as good as or better than prior 'known-good' results (the reference set).  (Note, if Freesurfer improves the results, then a new reference set is created, aka 'reset the standard').

=== Historical tests ===
The location of Buckner40 reference datasets with respect to previously released versions, and the development version, can be found at the following locations:

{{{
/space/freesurfer/subjects/test/buckner40/stable5_1_0
/space/freesurfer/subjects/test/buckner40/stable5_3_0
/space/freesurfer/subjects/test/buckner40/stable6
/space/freesurfer/subjects/test/buckner40/dev
}}}

And Adni60:

{{{
/space/freesurfer/subjects/test/adni60/stable5_1_0
/space/freesurfer/subjects/test/adni60/stable5_3_0
/space/freesurfer/subjects/test/adni60/stable6
/space/freesurfer/subjects/test/adni60/dev
}}}


=== Running the test ===

The 2 main scripts for running the Buckner40 and Adni60 tests are located here:

{{{
/space/freesurfer/subjects/test/scripts/comparison_test.sh
/space/freesurfer/subjects/test/scripts/comparison_utils.sh
}}}

To run the Buckner40 and generate groupstats, ssh to launchpad and run the {{{comparison_test.sh}}} as follows:

{{{
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/buckner40/dev \
     -generate_groupstats /space/freesurfer/subjects/test/buckner40/dev/groupstats_2016_07_30
}}}

To run the Adni60 and generate groupstats, call the scrpts the exact same way, just replace "Buckner40" in the path with "Adni60":

{{{
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/adni60/dev \
     -generate_groupstats /space/freesurfer/subjects/test/adni60/dev/groupstats_2016_07_30
}}}

=== Running tests as a cronjob ===

The Buckner40 and Adni60 datasets should be run on a weekly, monthly basis via cron, and should always be run (manually) as a precursor to a merge of the development branch with the master branch. To run these tests as a cronjob every week on Friday at midnight, the crontab file on launchpad should look as follows:

{{{
##########################################
## Run Buckner40 once per week (Friday, 11pm)
##########################################

FREESURFER_DEV=/usr/local/freesurfer/dev
BUCKNER40_DEV=/space/freesurfer/subjects/test/buckner40/dev
BUCKNER40_DEV_GROUPSTATS=$BUCKNER40_DEV/groupstats_$(date +"%Y-%m-%d")

00 23 * * 5 pbsubmit -f -m zkaufman -c "/space/freesurfer/subjects/test/scripts/comparison_test.sh -fs $FREESURFER_DEV -subjects_dir $BUCKNER40_DEV -generate_groupstats $BUCKNER40_DEV_GROUPSTATS"
}}}

=== Prelease procedure ===
In preparation for a release, a new folder should be created named for the expected freesurfer version number. For example, inpreparation of the release of freesurfer v7 we would create a "stable7" folder:

{{{
$> mkdir /space/freesurfer/subjects/test/buckner40/stable7
}}}

In that folder should be a {{{group.fsgd}}}, {{{raw_origs.tar.gz}}}, and {{{subjects.txt}}} files. These files already exist in the parent directory ({{{/space/freesurfer/subjects/test/buckner40}}}) and thus can be symlinked at follows:

{{{
$> cd /space/freesurfer/subjects/test/buckner40/stable7
$> ln -s ../buckner40.fsgd group.fsgd
$> ln -s ../buckner40_origs.tar.gz raw_origs.tar.gz
$> ln -s ../subjects.txt
}}}

The subjects data and groupstats can then be run continually using the freesurfer release candidates until it is finally run WITH THE ACTUAL FREESURFER VERSION TO BE RELEASED. The subject data and groupstats should be preserved for posterity with the exact version of freesurfer that is to be released:

{{{
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/stable7 \
     -subjects_dir /space/freesurfer/subjects/test/buckner40/stable \
     -generate_groupstats /space/freesurfer/subjects/test/buckner40/stable7/groupstats_2016_07_30
}}}