Freesurfer uses a tool called mritotal in the package called mni_autoreg [http://www.bic.mni.mcgill.ca/users/louis/MNI_AUTOREG_home/readme/].
It uses the MNI average_305 template volume to create the linear talairach affine transform. The transform file talairach.xfm is deposited in $subject/mri/transforms directory. The template volume average_305.mnc resides in /usr/pubsw/packages/mni/1.2/share/mni_autoreg directory in the MGH environment.
Because the mritotal method uses the correlation method, the resulting transform can be bad due to many reasons (The documentation for mritotal says that it works well for T1 dominant images). The last resort would be the manual registration, but that means that the variability of the transform from one person to next would become an issue.
The first step is to look at the result of mritotal (["recon-all"] leaves a mritotal.log in $subjid/scripts directory). At the end of this log, you will find a number printed:
Final objective function value = 0.06068489
If the number is bigger than 0.1, the transform is a suspect. If it is bigger than 0.2, it is wrong. My experience is that when this number is around 0.06, it is good.
You can see how good the transform is by loading the orig and applying a transform in tkmedit. Then compare the volume with $subjects_dir/average7/mri/mni305 volume (load it as aux volume). (Note that you cannot load average_305.mnc which has non 256^3 volume. tkmedit cannot handle two different sized volumes. The average7/mri/mni305 is conformed to have 256^3 so that you can load it under tkmedit.)
I have seen at least two causes of failures. One cause is the volume has the systematic grey scale variations or noisy white matter region. Another one is the volume which has a bright neck region.
In the case of grey scale problem, my suggestion is to try "T1" (white matter set to have 110 only) and then "brain". Here is the steps you take:
$ ["mri_convert"] T1 T1.mnc
$ mritotal -protocol icbm T1.mnc T1.xfm
Check the transform, using tkmedit. If still bad, try
$ ["mri_convert"] brain brain.mnc
$ mritotal -protocol icbm brain.mnc brain.xfm
Check the transform, using tkmedit.
If you find a good transform during the step above, replace mri/transforms/talairach.xfm with the good one. You must use the same name talairach.xfm.
When you see a big bright neck region, my option is to edit the volume to erase the bright neck region and run mritotal on the edited volume. It usually works by a simple editing, setting all the region to zero where y > certain value (around 170). This worked better than using "brain" volume.
If it is still bad, you can try using a manual registration tool to create the transform. Note that you have to create the MNI compatible talairach.xfm to go along with further freesurfer processes.
There exist a few manual registration tools.
The MNI GUI tool called register [http://www.bic.mni.mcgill.ca/software/register/register.html] allows you to get the linear transform manually by specifying the corresponding points and produces the xfm compatible with the process. The catch is that you have to convert your volume into MINC format, using ["mri_convert"] as shown above.
When you use ["tkregister"] or ["tkregister2"], you are advised to convert register.dat to the xfm, using ["regdat2xfm"].