FreeSurfer QA Tools (wiki page under construction)

The QA Tools are intended to be used in assessing the quality of one or more FreeSurfer recons. These scripts are capable of verifying all steps in the FreeSurfer recon-all stream were executed and in the correct order, as well as verifying all files exist and were created in the correct order. Additionally, these scripts can be used to detect potential outlier regions in the aseg.mgz within a dataset, calculate SNR and WM intensity values, and collect detailed snapshots of various volumes. The following people have been major contributors in developing these scripts: David Koh, Stephanie Lee, Jenni Pacheco, Vasanth Pappu, and Louis Vinke.

The QA Tools scripts are a work-in-progress. Any feedback, bug reports, or feature requests are appreciated (email vinke[at]nmr.mgh.harvard.edu).

Setup (NMR users)

Please email Louis at vinke[at]nmr.mgh.harvard.edu if you will be using the QA Tools so that we have an estimate of usage within the NMR center.

The most up-to-date version of the QA Tools scripts are located here:

/autofs/space/oceanic_001/users/vinke/QAtools_103111/data_checker

To use these scripts, first source your preferred version of FreeSurfer (i.e.):

source /usr/local/freesurfer/nmr-dev-env

Set your subjects directory and QA Tools scripts path:

setenv SUBJECTS_DIR /path/to/your/subjects/directory
setenv QA_TOOLS /autofs/space/oceanic_001/users/vinke/QAtools_103111/data_checker

Setup (non-NMR users)

Click the link below to download the QA Tools scripts. Extract all the contents of the tar file into a new directory. Source your local version of FreeSurfer, set your subjects directory variable, and set the QA Tools path:

source /path/to/local/freesurfer/
setenv SUBJECTS_DIR /path/to/your/subjects/directory
setenv QA_TOOLS /path/to/QA/Tools/scripts

Getting Started

Run the following command to view explanations for all the QA Tools options:

$QA_TOOLS/recon_checker -help

By default QA-Tools will check the status log, the order files were created, the aseg for any outliers, create snapshots of slices throughout various volumes, and calculate SNR and WM measures. See the flags below if you want to disable one or more of these options.

Arguments

Required Flagged Arguments

-s <subject1> [<subject2>...]

Case ID for one or more subjects located in $SUBJECTS_DIR (use -s-file as alternative)

Optional Flagged Arguments

-subjid <subject1> [<subject2>...]

Specify subject(s) to process

-s-file

Specify a file with a list of subjects. May be used instead of -s or -subjid

-snaps-out

Full path and name of snapshots HTML file. Default: "$SUBJECTS_DIR/QA/QA_check.html"

-snaps-detailed

Take a more detailed set of snapshots

-snaps-overwrite

Take all snapshots, overwriting any existing snapshots

-no-snaps

Do not take snapshots

-snaps-only

Only take snapshots

-outputFOF <file-order list>

Specify a file which lists the proper order of output files

-asegLUT <file>

Specify a file containing aseg outlier lookup table

-gen-outputFOF

Generate the file order list from the first subject

-gen-status

Generate the status file from the first subject

-gen-asegLUT <file>

Generate an aseg look up table from your subjects that can be used to identify aseg outliers

-nocheck-aseg

Do not check for aseg outliers

-nocheck-status

Do not check status log file for each subject

-nocheck-outputFOF

Do not check output order of files for each subject

-nocheck-SNR-WM

Do not calculate SNR and WM measures for each subject

-completeStatusFile <file>

Compares each subject's status log file to the specified log file

-completeStatusFiles <file>

Compares each subject's status log file to each of the complete status logs listed in <file>

-ignore <process1> [<process2> ...]

Specify one or more FreeSurfer processes to ignore while checking status files

!!! OLD INSTRUCTIONS FROM THIS POINT ON !!!

To check the output of one reconned subject

First you'll need to properly set your SUBJECTS_DIR. Also you'll need to set your QA_SCRIPTS directory. This should be the directory that you downloaded the QAtools directory. For our example here QAtools is in /space/tensor/17/users/jpacheco/QAtools.

setenv QA_SCRIPTS /space/tensor/17/users/jpacheco/QAtools

Now, from your SUBJECTS_DIR you are ready to run:

$QA_SCRIPTS/data_checker/recon_checker -nocheckasegoutliers -s bert -o QAsnap.html

where bert is your subject ID and QAsnap.html is the name of the output html file that you want to create.
For complete usage instructions please see the ReconChecker page.


What it does:


Outputs

There will be two log files output into your subjects scripts directory:

ls bert/scripts
recon_checker.bert.details.log
recon_checker.bert.summary.log

The recon_checker.bert.summary.log contains a summary of the status of that subject:

-------------------------------------------------------------
Running recon_checker on:
Thu Oct 12 14:51:24 EDT 2006
Checking Last Version Used:
/usr/local/freesurfer/dev/bin/recon-all
In output_file_checker
checking if output file order file is set...
output_file_order_file not specified, using default order
check to see if all files exist...
bert outputfiles_OK

In recon_all_status_log_checker
bert COMPLETE

This subject was run most recently using the dev version of FreeSurfer, all of the output files have been created in the correct order, and all the steps have been run. Another example of this log file is:

-------------------------------------------------------------
Running recon_checker on:
Thu Oct 12 14:54:24 EDT 2006
Checking Last Version Used:
/space/dijon/28/users/jpacheco/freesurfer_stable3_centos64//bin/recon-all
In output_file_checker
checking if output file order file is set...
output_file_order_file not specified, using default order
check to see if all files exist...
bert outputfiles_NOTOK

In recon_all_status_log_checker
bert STEPS_MISSING

This subject was run most recently using a version of FreeSurfer located /space/dijon/28/users/jpacheco/freesurfer_stable3_centos64, the output files are not in the ok order, and there are steps missing from the processing stream.

The recon_checker.bert.details.log will give more details as to what might be wrong if there are problems. For the second subject above, with steps missing, the recon_checker.bert.details.log looks like:

-------------------------------------------------------------
Running recon_checker on:
Thu Oct 12 14:54:24 EDT 2006
Checking Last Version Used:
/space/dijon/28/users/jpacheco/freesurfer_stable3_centos64//bin/recon-all
Checking ALL Versions Used:
/usr/local/freesurfer/dev/bin/recon-all,
/space/dijon/28/users/jpacheco/freesurfer_stable3_centos64//bin/recon-all,
In output_file_checker
SUBJECTS_DIR is /autofs/space/tensor_014/users/jenni

checking if output file order file is set...
output_file_order_file not specified, using default order
realOrder_plus1 is 72
/autofs/space/tensor_014/users/bert/mri/orig/001.mgz last modified 2005-11-02 12:53:09.000000000 -0500
/autofs/space/tensor_014/users/bert/mri/orig/002.mgz last modified 2005-11-02 12:53:56.000000000 -0500
ERROR: order is not commmon order: stopped at mri/transforms/talairach.xfm, should be mri/rawavg.mgz ...
real order:             common order:
..........              .........
..........              .........
..........              .........
/autofs/space/tensor_014/users/bert/mri/orig/001.mgz             /autofs/space/tensor_014/users/bert/mri/orig/001.mgz
/autofs/space/tensor_014/users/bert/mri/orig/002.mgz             /autofs/space/tensor_014/users/bert/mri/orig/002.mgz
mri/transforms/talairach.xfm            mri/rawavg.mgz
mri/rawavg.mgz          mri/orig.mgz
mri/orig.mgz            mri/nu.mgz
mri/nu.mgz              mri/transforms/talairach.auto.xfm
mri/transforms/talairach.auto.xfm               mri/transforms/talairach.xfm
..........              .........
..........              .........
..........              .........
bert outputfiles_NOTOK

In recon_all_status_log_checker
bert STEPS_MISSING
Expected to see the following steps before
#@# Tessellate rh Wed Jun 7 14:43:02 EDT 2006
in the recon-all-status.log file (line # 98):
--> Tessellate lh
--> Smooth1 lh
--> Inflation1 lh
--> QSphere lh
--> Fix Topology lh
--> Make Final Surf lh
--> Smooth2 lh
--> Inflation2 lh
--> Cortical ribbon mask lh
--> Sphere lh
--> Surf Reg lh
--> Contra Surf Reg lh
--> AvgCurv lh
--> Cortical Parc lh
--> Parcellation Stats lh
--> Cortical Parc 2 lh
--> Parcellation Stats 2 lh
However, these LH processes were executed previously.
Perhaps the RH output was modified and re-run without re-running the LH processes.

This log shows that the dev version of FreeSurfer was used for a previous set of processing, but most recently the private version was used. The output from the file order checking shows that the rawavg.mgz file was created after the transforms/talairach.xfm file but it was supposed to be created first. The file order checking was stopped there. The recon-all-status.log checker shows that all the LH processes are out of order. It offers a hint of explanation, saying that the RH may have been re-run without the LH since the LH processes were completed earlier. For this subject that was the case.

Lastly, an html file was created, QAsnap.html, and can be viewed in any standard browser. You can open it from the command line

mozilla $SUBJECTS_DIR/QAsnap.html

or put the path to your file directly in the browser. This will show you a web report of all the subjects you just ran. In our case we've only run one subject so we've only got one subject in our list
browser5.jpg

When you click on bert snapshots it will take you to a page with all the snapshots from that subject.
browser4.jpg
**NOTE: this shows just a portion of the snapshot output page, there are more snapshots if you scroll right and down.

Usage Recommendations

When you go through and check your outputs you can open the QAnotepad to keep track of your observations. A simple comand line to open the QAnotepad is:

$QA_SCRIPTS/QAnotepad -s <subject1> <subject2> ...

or, if you'd like to pass it your own config file then you can use:

$QA_SCRIPTS/QAnotepad -s <subject1> <subject2> ... -c <config file> -f <output file name>

The QAnotepad is a customizable tool that handles notes (both radio-button selections and free text) on various aspects of the FreeSurfer output, for further usage options and explanations, including how to design your own config file, see the QAnotepad wiki page.

To check the output of a group of reconned subjects

First you'll need to properly set your SUBJECTS_DIR. Also you'll need to set your QA_SCRIPTS directory. This should be the directory that you downloaded the QAtools directory. For our example here QAtools is in /space/tensor/17/users/jpacheco/QAtools.

setenv QA_SCRIPTS /space/tensor/17/users/jpacheco/QAtools

The first step is to make a aseg outliers look up table. You can run this command to do that:

$QA_SCRIPTS/data_checker/recon_checker -s <list of subjects> -makeAsegLookUp AsegLookUp.txt

where <list of subjects> is your own list of subjects, from which you want to make a look-up table. This will generate a file named AsegLookUp.txt, or whatever you have specified after the '-makeAsegLookUp' flag.

This command will generate an aseg look-up table, that can be used in subsequent runs. It will also check the versions used for each subject and check for outliers based on the table it has just created. The version and outlier information is saved into the summary and detail logs (see below for more information). It will NOT take snapshots or check any other ouptus, if you want to take snapshots, or check for other options as well you will need to add those additional flags to the command line. (see ReconChecker for more usage and options).

If you have already made your look-up table, but need to check for outliers again you can run the command:

$QA_SCRIPTS/data_checker/recon_checker -s <list of subjects> -checkasegoutliers -asegMeansFile AsegLookUp.txt

where <list of subjects> is your own list of subjects and AsegLookUp.txt is the name of your look-up table.

This command will use a previously made look-up table to check for aseg outliers. It will also check the versions used, the output files, the status log order, and take snapshots. If you want to skip some of these options you will need to add those additional flags to the command line. (see ReconChecker for more usage and options).

Outputs

Usage Recommendations

It's a good idea to make a seperate look-up table for each group in your study (i.e., in a study of patients versus controls make a look-up table for patients and another for controls) To do this you could run these commands:

$QA_SCRIPTS/data_checker/recon_checker -s $controls -makeAsegLookUp AsegLookUp_control.txt

$QA_SCRIPTS/data_checker/recon_checker -s $patients -makeAsegLookUp AsegLookUp_patient.txt

Once you have your group identified it's a good idea to rerun the -takesnapshots option. This will regenerate the opening html file, with a listing of all your subjects, making it easier to check the snapshots if needed.

To check the output of a group analysis

Future Directions

== notes ==

To-Do:

Things I've done:

* add a flag for -noasegoutlier - DONE by vp on 9/28/06

within the scripts set RECON_CHECKER_SCRIPTS to be $QA_SCRIPTS/data_checker/ - DONE by jp 10/12/06