Photo Reconstruction Documentation



If you use these tools in your analysis, please cite:

* Machine learning of dissection photographs and surface scanning for quantitative 3D neuropathology. H Gazula, H Tregidgo, B Billot, Y Balbastre, J Williams-Ramirez, R Herisse, LJ Deden-Binder, A Casamitjana, EJ Melief, CS Latimer, MD Kilgore, M Montine, E Robinson, E Blackburn, MS Marshall, TR Connors, DH Oakley, MP Frosh, SI Young, K Van Leemput, AV Dalca, B Fischl, CL Mac Donald, CD Keene, B Hyman, JE Iglesias. Under revision.)

Click here for a video with an overview of the method
video_screenshot.png



Purpose

This manual will provide step-by-step instructions on utilizing the Einscan Pro HD scanner, recommended preparation of tissue slabs after scanning, and outline the data processing pipeline to finalize a 3D reconstruction of brain slab images using a reference surface.


Einscan Turntable Protocol

This section will cover our standard operating procedure for scanning fixed tissue samples in the Massachusetts General Hospital (MGH). In the setup, we use an EinScan Pro HD scanner with the ‘industrial pack,’ which includes an additional turntable and tripod. A good surface scan is critical for accurately reconstructing the tissue slabs, as it is used as the reference to which the slabs are registered and deformed into. Later sections will cover surface mesh quality assessment and what to look for in a scan.

Scanner Setup

To set up the turntable and scanner, we adjust the tripod closely to the tissue while maintaining a clear view of the full turntable plate. Having the scanner not too far from the tissue will yield the best results. When using ‘Handheld’ mode, SHINING3D software will guide the user on the scanner distance. On the table, the scanner is adjusted to approximately 45˚, 8-10 inches from the turntable. Both the scanner and turntable are connected to the computer via USB and external power.

01_ScannerSetup.png
Scanner setup for turntable scan.

Next, we launch the most up-to-date Shining3D software for the Einscan Pro HD. Software for all EinsScan 3D scanners can be found at einscan.com/support/download/. Next, a menu will prompt a selection for the type of scan, fixed scan (using the turntable), handheld HD scan, or handheld rapid scan. We select “fixed scan” to use the turntable add-on and create a new “Non-texture Scan” project.

02_TextureSettings.png
Texture settings prompt when creating a new project.

At this point, the main scan menu should be on the screen, where we can customize different parameters for scan acquisition. On the left-hand side of the screen, there is a preview of the scanner’s field of view.

03_MainScanMenu.jpg
Main scan menu.

In the Scan Setting menu, ensure the “with Turntable” box is checked and the align mode is set to Features. The alignment mode “Turntable Coded Targets” is not used because tissue can slide as the turntable changes orientation. Sliding can be especially present when scanning the medial wall and can be helped by gently patting the tissue with a linen towel to remove some excess fixative on the tissue. When using the handheld mode, we highly recommend using markers which can vastly improve scanning speed and user experience.

Turntable steps determine how many scans will be acquired and stitched for this session. Higher numbers can lead to more accurate data but can increase scan time. We set turntable steps to 48, which we consider a good balance between acquisition time and scan quality.

For handheld scans, we use the parameters:


Sample Setup

We place a glass sheet atop to protect the turntable and ease cleanup. The glass sheet contains markers that can aid tracking in handheld scans but will not be used with the feature alignment mode. The sample is placed in the middle of the turntable so that the scanner gets the best view of the sample as it rotates. Occluding the turntable markers will also affect the stitching of scans if the scan is set to “Turntable Coded Targets.” When doing a handheld scan, make sure the sample is not covering the placed markers on the glass sheet. Note the sample does not include the cerebellum.

05_SamplePlacement.png
Sample placement on the turntable with sample lateral side down.

Two scan projects are created for all samples on opposite sides. For hemispheres, we first acquire a scan with the sample medial side down. Next, the hemisphere is flipped to be lateral side down, and the tissue is more likely to shift when the turntable steps. To reduce sample movement, we wipe down excess fixative on the sheet of glass to improve traction. Whole Brains are scanned upright (brain stem down) and upside down. Although these types of samples do not tend to move during the scan, they can deform between being upright versus upside down, which can lead to issues aligning the two sets of point clouds.

Tip: Samples should have the cerebellum removed before acquisition. The cerebellum is not considered in the reconstruction, including it present in the scan will lead to incorrect slice reconstruction.

06_ScanIncorrect.png
Scan incorrectly including the cerebellum.

Handheld Scanning

Handheld mode is also an option, particularly for fresh tissue, due to the tissue being unable to maintain its shape. It cannot be flipped for two-sided scans. In this mode, a transparent surface can be used to scan both sides of the sample without having to shift the easily deformable tissue.

07_ExampleHandheld.png
Example of a handheld scan setup.

Running the Scan

In the scan workspace, the preview window is used to center the tissue sample in the camera view. Set the red crosshairs to the middle of the sample for full coverage of the tissue as the turntable moves.

10_expsure_good.png
Scanner preview.

The camera exposure has a large effect on scan quality. Incorrect exposure can lead to noisier surfaces in the resulting scan point clouds. Exposure is adjusted in the main scan menu on the slider under the camera preview screen.

09_ScannerExp.png
Scanner exposure slider.

To determine if the sample is not exposed appropriately, the preview window will show bright regions with red. The slider is set to an exposure where there are some but not many red regions. To avoid any issues with exposure, sometimes we use HDR mode, which can be set in the Scan Settings menu for fixed turntable scans.

11_exposure_over.png

12_exposure_under.png

10_expsure_good.png

Overexposed (red filter on sample + bright image)

Underexposed (black filter on sample + dark image)

Sufficient exposure (can see sample, markers, decent contrast)


HDR Mode

HDR mode is used for samples with dark features like bruising or veins. The scanner can have a hard time picking up darker features on the sample. HDR mode will take three images at each turntable step at different exposures to capture darker and deeper structures in the tissue (e.g., sometimes this mode is good if point clouds aren’t generated near medial structures, including 3rdventricle, septum pellucidum, corpus callosum, parahippocampal gyrus, occipitotemporal gyrus, and any optic structures – nerve, chiasm)

13_dark_veins.png
Sample with dark veins.

Editing and Saving Data

Removing Excess Elements

  1. Remove excess data or turntable markers using the deleting tool in case the scanner picks up other neighboring geometry that’s not brain – though this is not likely on a fixed scan.
  2. Select extra elements by holding the shift key + left click.
  3. Delete these points by clicking the “Delete Selected Data” button or the delete key.
  4. Press the “Complete Editing” check-mark icon once all necessary edits are done.
  5. Note: Editing point clouds is easier than editing meshes on the software.

14_UI_guide.png
Performing the second run after removing excess elements from the first set of point clouds.

  1. When finished editing, an optimized point cloud will be generated, which should have saved automatically as “Project 1.”
  2. Briefly QA scan data, looking at noise or holes in the surface of the scan. Some gaps in the point clouds are normal. Any scan can be redone to get better results at any time.


Aligning both runs

  1. If you have two scanning sessions, you can align the cloud points using three reference regions.
  2. Click the alignment icon and drag and drop both scanning sessions to the float and fixed windows.
  3. Hold the shift key and left click to select three collinear points on each scan session. Ideally, we would want to place one point on the anterior, superior, and posterior sides of the brain.
  4. Click Complete when finished with the alignment.


Creating a Mesh

  1. Click on the mesh icon to generate a mesh model from your generated point clouds.
  2. Select watertight.
  3. Choose the “high-detail” option for the best mesh resolution.
  4. This reconstruction should take approximately 20+ minutes, depending on the equipment.
  5. Save your mesh when reconstruction has finished. Your mesh should be saved as a .stl file automatically, but it is also recommended to save your point clouds (.asc files).


Addressing mesh quality

When acquiring the mesh, the sides of the sample are critical for a good point cloud alignment. A bad alignment between scans can cause a lot of issues, leading to scans that are not representative of the tissue. The following example shows a handheld scan where the sides of the tissue were not acquired. This can lead to a seemingly flatter sample and will affect the quality of the final reconstruction.

15_bad_mesh.png
Mesh missing sides of the top scan.

Ideally, we want the sides of the mesh to be filled like in the following example. When acquiring a handheld scan, it can be difficult to acquire the sides of the sample without acquiring other surfaces. These additional surfaces can be edited and deleted in the same way as markers are deleted.

16_good_mesh.png
Mesh correctly including the sides of the tissue.

Note that the previous surface has some holes due to deep sulci, which should not be an issue with the reconstruction. Watertight meshing should be able to handle small holes.


Preparing Slab Data

The following guide should be applied for the preparation of the slab images. This step should be completed prior to any GUI usage.

Guidelines for Brain Slab Image Acquisition

Before processing slice segmentation and volume reconstruction, there are some important requirements for the photographic setup of the brain slabs.

Use a high-contrast background that stands out in uniform contrast to the brain slabs, like a solid black background, so slabs are easier to segment downstream with a simple threshold. Using a complex background with color similar to your brain slabs, will lead to a more manual intervention like having to manually trace contours, which is not only time-consuming but also non-reproducible.

17_low_constrast.png

18_high_contrast.png

Difficult to segment low contrast background.

Easy to segment high contrast background.

Additionally, all tissue block faces must be facing the same direction. All slices should be either from an anterior or a posterior view. Note if slabs are from the posterior view, an extra flag,

--photos_of_posterior_side must be used for the final reconstruction.

When a single slice produces multiple tissue blocks, they should be placed as similar to the original anatomy as possible. For example, after acquiring slices from the frontal pole, the initial blocks corresponding to the temporal lobe will often be detached from the main tissue block. The following figure a) shows a good tissue placement consistent with the anatomy while b) shows too much distance between the blocks of tissue that is not anatomically accurate.

19_slab_placement.png
Tissue slab placement a) Properly placed tissue b) Tissue placed in a non-anatomical manner.

Another example of difficult tissue placement can be the corpus callosum. In the following figure, we show a great example slab image with a critical flaw. Fiducials are clearly shown, and there is high contrast for the background and great lighting. However, there is an issue with tissue placement. Note the rightmost slab has an incorrect placement of the corpus callosum, which will affect estimates of the medial vessel.

20_incorrect_tissue_placement.png

Markers for pixel and perspective corrections

To calibrate pixel size, we will need landmarks with known distances in between them to be provided by the user. The minimum requirement is a ruler, which provides us with two discrete landmarks. This enables rough This method is preferable for legacy images that only have a ruler for reference.

21_twoPoint.jpeg

Tip: It is preferable to use two points that are as far apart as possible. This is because the relative error in pixels/mm is minimized by choosing two points with a greater distance between them.

A better photographic setup is to have four landmarks in the corners of a rectangle of known dimensions. This enables us to correct for both pixel size and perspective. For example, in the picture below, we can easily tell that the 4 points marked on the grid define a rectangle of dimensions 17x20 cm.

22_fourPoint.png

The most optimal scenario is to have 4 points defined by highly salient fiducial markers. This enables our software to automatically recognize the fiducials, which minimizes human interaction, reducing time and maximizing reproducibility. We strongly recommend that you print these set fiducials and place them on the corners of the board.

23_fiducials.png

Tip: For the greatest benefit, these fiducials should be attached to a small piece of rubber/wood/plastic that lifts them X mm above the board, where X is the thickness of your typical brain slice. Lifting the fiducial minimizes error due to the surface of the brain slice NOT being at the same distance as the plane of the board surface, but rather a few millimeters above.

24_fiducialTissue.png

The fiducial markers serve as the four corners of our corrected image; hence all tissue should be within the rectangle created from the four fiducial corners. The following image shows slab photos where the fiducials were not placed sufficiently apart.

25_tissueSpace.png

Additionally, by drawing a rectangle with all fiducial corners, it is clear that much of the tissue is outside of the pixel-corrected region.

26_tissueBox.png

In sum, here is an example of what an optimal photographic setup would look like, including the uniform, high-contrast background and provided fiducials located at known distances.

27_goodContrast.png

Tip: It does not matter that these slabs are not presented on a grid. As long as you know the distance in millimeters between the fiducials, our software will draw digital rulers for your images later.

Cloning The Freesurfer Development Git Repo

Additionally, download the development version of Freesurfer by cloning the dev branch here:

https://github.com/freesurfer/freesurfer

Creating a Directory Structure

We recommend the following directory structure for data and case organization. Each case should have a directory. Inside your case directory, create seven additional directories to store the following outputs.

  1. photos – Brain slab images (.JPG, .PNG).
  2. deformed - Pixel-corrected images.
  3. point_clouds – Files from the project collected from the surface scanner.
  4. mesh – Watertight mesh processed from point clouds on the scanner (.stl).
  5. masked – Binarized masks created from pixel-corrected photos.
  6. connected_components - The outputs from the connected components GUI.
  7. recon - The final reconstructed image volumes.

After each step, ensure that you are uploading your data back to your subject’s main directory. See below how files are divided between each of the sub-directories.

28_demoDirectory.png

29_demoDirectory.png
29.2_demoDirectory.png


Image Correction

The following guide should be applied for pixel correction of the brain slab image data. These GUIs will use images of raw brain slab photographs as input and creates deformed, pixel-corrected images. By selecting fiducials of known distance from each other, we can correct for non-uniformities in pixel size across the image.

Correction with fiducials

For images with fiducials, a calibration will be made before pixel correcting the images. To generate a calibration, a tissue-free image must be used.

  1. Source the dev version of Freesurfer.
  2. Run the fiducial calibration with the command “calibration_with_fiducials.”

30_fiducialCalibIntro.png

After loading the image, we can start the calibration process.

31_fiducialCalibWindow.png

We can use pan and zoom to have a precise selection in these GUIs. First, select the center of one of the fiducials, followed by selecting the outer ring.

32_fiducialCenter.png

33_fiducialDiameter.png

The first selection at the fiducial center.

The second selection at the outer circle of the fiducial.

After selecting all the points, the width and height between the fiducial markers. After clicking “Perform calibration,” a ‘.npz’ calibration file is saved.

Next, run the command “fiducial_correction” and add the relevant directories to correct all images automatically.

34_fiducialCalibDirectory.png

Retrospective Registration

Although less preferable, for images that do not have fiducials, we can still pixel-correct images using other references in the image, like a ruler, for each of the images.

  1. Source the dev version of Freesurfer.
  2. Organize your directory structure.
  3. Run the retrospective registration command “retrospective_correction.”

Select “Click to Start” on the first dialogue box.

35_retrospectiveIntro.png

Select mode and indicate file paths. A new dialogue box (pictured below) will appear.

Follow the directions as such:

  1. Select the directory with the input images (the untouched sectioned brain images) by clicking the Choose Folder option.
  2. Select the directory for the output images to be saved. As stated, this directory must already be created.
  3. Select the mode that will be used for pixel correction. Select a mode that is compatible with your image setup. 2 points should be chosen if there is a single axis of ground truth measurement (i.e., One ruler in the image), and four should be chosen if there are two axes.

36_retrospectiveDirectory.png

37_retrospectiveTissue.png

A new interactive window with your brain slabs and a “Ruler Length” option will be displayed. Click visible markers that provide ground truth measurement. Use these as the basis for your ruler Length, which should be entered in millimeters. Then, click register and complete the remaining images. The pixel-corrected images will then populate your output directory. Finally, save your subject’s registered images to your subject’s ‘deformed’ subdirectory.


Binarizing Images

Binary masks are created to only include the tissue slice and not include any cortex. Image editing software like GIMP can be used to create these binary masks. A combination of the thresholding and lasso tool is used to remove any non-slice tissue and the background.


Connected Components

This section covers how to use the connected components GUI to select brain slices.

  1. Source the dev version of Freesurfer.
  2. Run the fiducial calibration with the command “connected_components.”

Following these directions, select the directory with the input images (the pixel-corrected deformed images).

  1. Select the directory with the input masks created through GIMP.
  2. Select the directory where you want the output masks saved.

By clicking Click to Start, you will land on the following interactive module. Input the directory destinations here.

38_ccIntro.png

39_ccDirectory.png

By clicking Start Processing, you will open a new interactive window displaying the deformed brain slab photographs. For each image, select slices in order by clicking and dragging to draw a box in a region of each individual slab in the photograph. After a box has been drawn in each individual slab, click Process Image to save the output .npy files. Make sure no boxes are overlapping.

40_ccMain.png

Once a box is drawn over the tissue, the slab will be colored to show what is being included as a slice.

41_ccBox.png

Above is an example of how to correctly draw the boxes within the brain slabs themselves. These boxes inform the GUI that the pixels connected to these slabs are all related. It is important that there is a disconnection between these slabs with some empty/black space between them so the GUI does not count two separate slabs as a single connected component. Additionally, note that the slabs are selected in the order they were cut.

42_ccBoxLarge.png

Above is an example of how NOT to draw boxes for this GUI. Notice the occurrence of overlap between boxes and the inclusion of an abundance of black space. Try to keep the boxes drawn to small regions within the brain slabs themselves.

IMPORTANT: The cortical surface should not be part of the masks, so the first (if photographed from the anterior side) or last slice (if photographed from the posterior side) should NOT be considered. You can either leave it out of the mask or ignore it when processing with the unknown GUI.


Photo Reconstruction

This is a guide to submitting the final outputs from the photo-calibration aspect of the processing pipeline for reconstruction and surface estimation.

To begin, make sure to have the dev version of Freesurfer installed.

Next, we can call the scrips to perform the slice reconstruction. Use the following command for help with any flag for the script:

mri_3d_photo_recon -h

--input_photo_dir

Directory with input pixel-corrected photos.

--input_segmentation_dir

Directory with input slab masks/segmentations

--ref_mask

When using a binary volume as a reference.

--ref_surface

Using a 3D surface scan as a reference.

--ref_soft_mask

Using the provided average atlas. Best for using retrospective processing on data without a better reference.

--mesh_reorient_with_indices

Vertex indices of the frontal pole, occipital pole, and top of the central sulcus, separated with commas, for mesh alignment.

--photos_of_posterior_side

Use when photos are taken of the posterior side of slabs (default is anterior side).

--order_posterior_to_anterior

Use when photos are ordered from posterior to anterior (default is anterior to posterior).

--allow_z_stretch

Use to adjust the slice thickness to best match the reference. You should probably *never* use this with soft references (ref_soft_mask).

--rigid_only_for_photos

Switch on if you want photos to deform only rigidly (not affine).

--slice_thickness

Slice thickness in mm.

--photo_resolution

Resolution of the photos in mm.

--output_directory

Output directory with reconstructed photo volume and reference

In the bottom Freeview menu, we can find the number corresponding to each of the vertices of each of these areas.

43_vertexFP.png

44_vertexOP.png

45_vertexCS.png

Frontal Pole

Occipital Pole

Top of the central sulcus

These are used for the --mesh_reorient_with_indices flag, where indices are comma-separated in the order of frontal pole, occipital pole, and top of central sulcus. When using this flag, it will look something like:

Example for running reconstruction
mri_3d_photo_recon \




Photo-Synthseg Segmentation

The 3D reconstructed stacks can be segmented with SynthSeg using the flag --photo. Please see the wiki page for SynthSeg for further information.

PhotoTools (last edited 2023-07-26 16:36:28 by JonWilliamsRamirez)