Differences between revisions 36 and 37
Deletions are marked like this. Additions are marked like this.
Line 46: Line 46:
 * ADNI-HIPPO - [[http://www.hippocampal-protocol.net/SOPs/index.php|135 ADNI subjects with manual segmentations of hippocampus]]  * ADNI-Hippo - [[http://www.hippocampal-protocol.net/SOPs/index.php|135 ADNI subjects with manual segmentations of hippocampus]]
Line 107: Line 107:
 * AH: create ADNI-HIPPO data set, composed of 135 subject with manually labeled hippo
 * AH: create a test script for the ADNI-HIPPO subject which shows hippo dice scores.
 * AH: create ADNI-Hippo data set, composed of 135 subject with manually labeled hippo
 * AH: create a test script for the ADNI-Hippo subject which shows hippo dice scores.

This page is readable only by those in the LcnGroup.

Author(s): Nick Schmansky, Andrew Hoopes

See: Samseg

Samseg Aseg Testing

This page documents one front of the work on Samseg, where testing of a version of Samseg operating on T1-weighted input and using the existing RB FS atlas, is conducted to compare its performance against the existing FreeSurfer v6.0 subcortical segmentation processing stream. Initial work was conducted in an evaluation task: SamsegEvaluationMarch2016. The current work pertaining to this page extends that work by:

  • Creation of a test cycle allowing the comparison of a subject run of Samseg (which is under algorithmic development) to the FS aseg results from v6 for that subject. The test cycle should generate reports on a run, allowing for debug, and evaluation of performance to meet, initially, the aims of the CorticoMetrics grant aims (ie, an accelerated FS based on T1 input only)

  • Creation of a data set which includes enough subjects to cover the wide variety of possible inputs. Starting with Buckner40 and ADNI60, and continuing with many others, particularly subjects having lots of neck in the FOV, as well as 'tough cases' both in terms of registration and segmentation, plus a variety of scanner model, and age and disease states.
  • Documentation of design decisions.

Those working on this project include: Doug Greve (DG), Nick Schmansky (NS), Andrew Hoopes (AH), Christian Larsen (CH) and Lee Tirrell (LT)

Test Data

structure:

The test subjects and scripts are located here: /cluster/fsm/users/samseg

samseg
├── scripts
├── subjects
│   ├──ADNI60
│   ├──Bucker40
│   └── ...
└── tests
    ├──testdate_ADNI60
    ├──testdate_Bucker40
    └── ...

Test sets used to compare samseg results will include:

  • Buckner39 - 39 manually labeled subjects used to create the FS subcortical atlas
  • Siemens13 - 13 manually labeled subjects scanned on Siemens Sonata (currently used in testing the FS atlas: AsegTestNotes)

  • GE14 - 14 manually labeled subjects scanned on GE Signa (currently used in testing the FS atlas: AsegTestNotes)

  • Buckner40 - public buckner40, processed by FS v6, where aseg's have been manually inspected as part of FS release testing
  • ADNI60 - processed by FS v6, where aseg's have been manually inspected as part of FS release testing
  • ADNI-Hippo - 135 ADNI subjects with manual segmentations of hippocampus

  • ADNI-1.5T-FS-5.3 - 1.5T ADNI data processed by FS 5.3, and QA'd by Tian Ge (but asegs not each manually inspected)
  • ADNI-3T-FS-5.3 - 1.5T ADNI data processed by FS 5.3, and QA'd by Tian Ge (but asegs not each manually inspected)
  • ADNI-1.5T-all - all 1.5T data from ADNI, 001.mgz files only
  • ADNI-3T-all - all 3T ADNI data, 001.mgz files only
  • Outliers - consisting of subject scans known to be problematic (lots of neck, skewed placement in scanner, etc.)
  • IXI, HCP, OASIS, HABS, BRAINS, Mindboggle-101, BGSP - all publicly available datasets where the intent is to process by FS v6 to include in the aseg test set.
  • ANDI-Long - test-retest set
  • Bammer120 - test-retest set
  • disorder sets - data from various disorder studies (outside of AD), eg. ms, schizophrenia, autism, tumor, child, addiction, etc.

running a test:

To run a test, use the runtest script located in /cluster/fsm/users/samseg/scripts. To test samseg with a multi-subject set, use the -set flag and indicate the test set directory (located in /cluster/fsm/users/samseg/subjects) as well as a test results output dir:

./runtest -all -set <subjects dir> <test outdir>

example: scripts/runtest -all -set subjects/ADNI60 tests/newADNItest

-all runs each test step, including running samseg, computing dice scores, and creating charts. To specify which steps to run, use -samseg, -dice, and/or -chart instead. To test samseg with an individual subject, use the -ind flag and indicate the path to the subject as well as a test results output dir:

./runtest -all -ind <subject> <test outdir>

NOTE: I was under the impression that matplotlib (python plotting package) was installed for the center, but I guess not. Instead, I've pointed the runtest shebang to my anaconda install on topaz

results:

In the test output directory, the script will create a dir for each subject and an analysis dir (not produced for an individual subject test). Each subject contains its samseg output as well as dice.dat and dice.log, produced by mri_compute_seg_overlap. In the analysis dir, subjs_dice.log is a summary file of the mean overlap for each subject. labels_dice.log is a summary file of the mean overlap for each brain structure across all subjects. The runtest script also creates a labels_dice.no_outliers.log, which ignores any subjects with a mean overlap below a certain threshold (default is 0.2 - can be changed with -outlier); the subjects ignored are written to outliers. An associated chart is created for each of these three files and saved as .png.

Test Runs

Timing test:

  • 03.25.2017 - 2 Buckner39 subjects - average elapsed time 1256 s (20.9 min) - 2017-03-25_timing


After adding Rician noise, includes comparison to v6 aseg:

  • 03.25.2017 - Siemens13 - all subjects successful (mean: 0.83, overall: 0.87) - 2017-03-25_Siemens13
    03.25.2017 - GE14 - registration fixed - all subjects successful (mean: 0.77, overall: 0.83) - 2017-03-25_GE14
    03.25.2017 - Buckner39 - no change - 21 subjects with unsuccessful registration - 2017-03-25_Buckner39
    03.25.2017 - Buckner40 - registration fixed - all subjects successful (mean: 0.86, overall: 0.89) - 2017-03-25_Buckner40
    03.25.2017 - ADNI60 - no change - 19 subjects with unsuccessful registration


V6 aseg:


March 2017 initial test:

  • 03.19.2017 - Siemens13 - all subjects produced dice scores above 0.8! - 2017-03-19_Siemens13
    03.19.2017 - GE14 - 2 subjects had unsuccessful registration - 2017-03-19_GE14
    03.19.2017 - Buckner39 - 21 subjects had unsuccessful registration! - 2017-03-19_Buckner39
    03.17.2017 - Buckner40 - 2 subjects with low dice due to poor reg, and 1 subject failed - 2017-03-17_Buckner40
    03.06.2017 - ADNI60 - 19 subjects with low dice due to poor registration - 2017-03-06_ADNI60

Tasks

  • Fix registration issue
  • AH: create ADNI-Hippo data set, composed of 135 subject with manually labeled hippo
  • AH: create a test script for the ADNI-Hippo subject which shows hippo dice scores.

SamsegTesting (last edited 2021-04-20 18:17:35 by DevaniCordero)