Differences between revisions 11 and 12
Deletions are marked like this. Additions are marked like this.
Line 17: Line 17:
The location of Buckner40 reference datasets with respect to previously released versions can be found at the following locations: The location of Buckner40 reference datasets with respect to previously released versions, and the development version, can be found at the following locations:
Line 23: Line 23:
/space/freesurfer/subjects/test/buckner40/dev
Line 25: Line 26:
Monthly/weekly testing for the development branch should be stored here: And Adni60:
Line 28: Line 29:
/space/freesurfer/subjects/test/buckner40/stable5_1_0
/space/freesurfer/subjects/test/buckner40/stable5_3_0
/space/freesurfer/subjects/test/buckner40/stable6
Line 44: Line 48:
ssh launchpad
cd /space/freesurfer/subjects/test/scripts
comparison_test.sh \
  -fs /usr/local/freesurfer/dev \
  -subjects_dir /space/freesurfer/subjects/test/buckner40/dev \
  -generate_groupstats /space/freesurfer/subjects/test/buckner40/dev/groupstats_2016_07_30
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/buckner40/dev \
    -generate_groupstats /space/freesurfer/subjects/test/buckner40/dev/groupstats_2016_07_30
Line 51: Line 54:

To run the Adni60 and generate groupstats, call the scrpts the exact same way, just replace "Buckner40" in the path with "Adni60":

{{{
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/adni60/dev \
     -generate_groupstats /space/freesurfer/subjects/test/adni60/dev/groupstats_2016_07_30
}}}

=== Running tests as a cronjob ===

The Buckner40 and Adni60 datasets should be run on a weekly, monthly basis via cron, and should always be run (manually) as a precursor to a merge of the development branch with the master branch. To run these tests as a cronjob every week on Friday at midnight, the crontab file on launchpad should look as follows:

{{{
##########################################
## Run Buckner40 once per week (Friday, 11pm)
##########################################

FREESURFER_DEV=/usr/local/freesurfer/dev
BUCKNER40_DEV=/space/freesurfer/subjects/test/buckner40/dev
BUCKNER40_DEV_GROUPSTATS=$BUCKNER40_DEV/groupstats_$(date +"%Y-%m-%d")

00 23 * * 5 pbsubmit -f -m zkaufman -c "/space/freesurfer/subjects/test/scripts/comparison_test.sh -fs $FREESURFER_DEV -subjects_dir $BUCKNER40_DEV -generate_groupstats $BUCKNER40_DEV_GROUPSTATS"
}}}

Buckner40 Testing

see also: TestPlan, and Buckner40Notes

Introduction

The 'Buckner40' is a name given to a small subset of subjects originating from Washington University, collected by Randy Buckner and colleagues. It consists of a group of 40 nondemented subjects, ranging in age from 18 to 93 (with no subjects between 30 to 65 years of age), 21 women and 19 men. These data are a subset of a larger open-access structural data set that can be obtained from the fMRIDC at: http://www.fmridc.org/f/fmridc, accession number 2-2004-1168X.

For Freesurfer, this dataset is especially nice as a test-set because of the wide subject age range, and the quality of the scans, which, while good, are not great, but are typical of what Freesurfer has to handle.

The Buckner40Notes wiki page documents the work of examining these brains, slice by slice, to verify their correctness, in terms of how well Freesurfer has reconstructed the white and pial surfaces, and segmented the subcortical structures.

It is important that Freesurfer's recon-all processing stream continue to process all of the subjects successfully, where that means the results are as good as or better than prior 'known-good' results (the reference set). (Note, if Freesurfer improves the results, then a new reference set is created, aka 'reset the standard').

Historical tests

The location of Buckner40 reference datasets with respect to previously released versions, and the development version, can be found at the following locations:

/space/freesurfer/subjects/test/buckner40/stable5_1_0
/space/freesurfer/subjects/test/buckner40/stable5_3_0
/space/freesurfer/subjects/test/buckner40/stable6
/space/freesurfer/subjects/test/buckner40/dev

And Adni60:

/space/freesurfer/subjects/test/buckner40/stable5_1_0
/space/freesurfer/subjects/test/buckner40/stable5_3_0
/space/freesurfer/subjects/test/buckner40/stable6
/space/freesurfer/subjects/test/buckner40/dev

Running the test

The 2 main scripts for running the Buckner40 and Adni60 tests are located here:

/space/freesurfer/subjects/test/scripts/comparison_test.sh
/space/freesurfer/subjects/test/scripts/comparison_utils.sh

To run the Buckner40 and generate groupstats, ssh to launchpad and run the comparison_test.sh as follows:

$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/buckner40/dev \
     -generate_groupstats /space/freesurfer/subjects/test/buckner40/dev/groupstats_2016_07_30

To run the Adni60 and generate groupstats, call the scrpts the exact same way, just replace "Buckner40" in the path with "Adni60":

$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/adni60/dev \
     -generate_groupstats /space/freesurfer/subjects/test/adni60/dev/groupstats_2016_07_30

Running tests as a cronjob

The Buckner40 and Adni60 datasets should be run on a weekly, monthly basis via cron, and should always be run (manually) as a precursor to a merge of the development branch with the master branch. To run these tests as a cronjob every week on Friday at midnight, the crontab file on launchpad should look as follows:

##########################################
## Run Buckner40 once per week (Friday, 11pm)
##########################################

FREESURFER_DEV=/usr/local/freesurfer/dev
BUCKNER40_DEV=/space/freesurfer/subjects/test/buckner40/dev
BUCKNER40_DEV_GROUPSTATS=$BUCKNER40_DEV/groupstats_$(date +"%Y-%m-%d")

00 23 * * 5 pbsubmit -f -m zkaufman -c "/space/freesurfer/subjects/test/scripts/comparison_test.sh -fs $FREESURFER_DEV -subjects_dir $BUCKNER40_DEV -generate_groupstats $BUCKNER40_DEV_GROUPSTATS"

Buckner40Adni60Testing (last edited 2023-07-06 17:55:26 by buildqa)