Post-acquisition image based compensation for thickness variation in microscopy section series

11/25/2014
by   Philipp Hanslovsky, et al.
0

Serial section Microscopy is an established method for volumetric anatomy reconstruction. Section series imaged with Electron Microscopy are currently vital for the reconstruction of the synaptic connectivity of entire animal brains such as that of Drosophila melanogaster. The process of removing ultrathin layers from a solid block containing the specimen, however, is a fragile procedure and has limited precision with respect to section thickness. We have developed a method to estimate the relative z-position of each individual section as a function of signal change across the section series. First experiments show promising results on both serial section Transmission Electron Microscopy (ssTEM) data and Focused Ion Beam Scanning Electron Microscopy (FIB-SEM) series. We made our solution available as Open Source plugins for the TrakEM2 software and the ImageJ distribution Fiji.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

02/01/2020

Estimation of Z-Thickness and XY-Anisotropy of Electron Microscopy Images using Gaussian Processes

Serial section electron microscopy (ssEM) is a widely used technique for...
04/29/2019

Convolutional nets for reconstructing neural circuits from brain images acquired by serial section electron microscopy

Neural circuits can be reconstructed from brain images acquired by seria...
06/30/2017

Adversarial Image Alignment and Interpolation

Volumetric (3d) images are acquired for many scientific and biomedical p...
05/07/2018

Synaptic Cleft Segmentation in Non-Isotropic Volume Electron Microscopy of the Complete Drosophila Brain

Neural circuit reconstruction at single synapse resolution is increasing...
12/07/2017

Parallel Prefix Algorithms for the Registration of Arbitrarily Long Electron Micrograph Series

Recent advances in the technology of transmission electron microscopy ha...
04/27/2020

Reconstructing normal section profiles of 3D revolving structures via pose-unconstrained multi-line structured-light vision

The wheel of the train is a 3D revolving geometrical structure. Reconstr...

Code Repositories

z-spacing

Estimate the correct positions of sections of three dimensional image data.


view repo

em-thickness-estimation

Estimate thickness of slices (or at local points) of EM data.


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Serial section microscopy has been used to study the anatomy of biological specimens for over 130 years [1]

. Today, neuroscientists use serial section microscopy to reconstruct the microcircuitry of animal nervous systems in their entirety. Electron microscopy (EM) offers the necessary resolution to resolve individual neuronal processes and synapses

[2]

. However, both serial section transmission electron microscopy (ssTEM), and block-face scanning electron microscopy (BF-SEM) generate section series with significant thickness variance. In a situation where the size of relevant structures is close to the resolution-limit imposed by section thickness, such variance can render accurate reconstruction impossible.

LF 1 top row shows an example of this thickness variance in a virtual -cross-section through a section series acquired with a focused ion beam milling scanning electron microscope (FIB-SEM).

Figure 1: FIB-SEM series before (top) and after (bottom) section thickness correction. (a) shows an -slice through the volume, (b) shows the corresponding pairwise similarity matrix (PSM) with intensity-encoded normalized cross-correlation (NCC) values in the same -coordinate frame. Arrows mark a section range that appears highly compressed in the original acquisition and biologically plausible after correction.

In order to rectify these distortions, we have developed an image based method that estimates the position of each individual section along the -axis of the volume. Using this information, a corrected volume can be rendered, greatly facilitating manual and automatic reconstruction efforts.

The underlying observation of our method is that the biological tissue is a signal whose changing speed varies slowly across the section series. Sections that are closer to each other generally look more similar than those farther apart. In addition, individual sections vary in image quality and may contain preparation artifacts which contributes uncorrelated noise to the pairwise similarity estimate. We therefore simultaneously estimate the decay of simple pairwise similarity measures as a function of the distance between two sections, the quality of each individual section, and the optimal position of each section within the series such that the updated pairwise similarity measure is smooth. The result is a unique -position for each section, compensating for differences in thickness (see LF 1 bottom row). This estimate can be executed locally, accounting for varying section thickness across the section plane. We make our method available as plugins for the TrakEM2 software [3], and the ImageJ distribution Fiji [4], and have published the source code on GitHub111https://github.com/saalfeldlab/em-thickness-estimation.

2 Related Work

The problem of varying thickness in microscopy section series has been observed and addressed in previous work. De Groot [5] reviews four different methods to estimate section thickness: (i) measuring tissue folds that are twice as thick as the section, (ii) comparing electron scattering in the section with a standard test curve, (iii) estimating section thickness from interference patterns that occur when splitting a light beam and placing the transparent refractile Epon-section in one of the otherwise identical beam paths, and (iv) re-embedding the sections, re-sectioning them perpendicularly to the -plane, and then measuring the section thickness within these sections from subsequent EM. Method (i) depends on an artifact that compromises image quality, options (ii,iii) require specialized instruments, and option (iv) is not only laborious but requires that sections can be re-embedded and re-imaged which is not possible in destructive imaging modalities such as BF-SEM.

Berlanga et al. [6] manually annotate the top and bottom surface areas of small image stacks and describe these surfaces by polynomial functions. They then transform the image density (RGB-channel values) such that both surface areas are flat and perpendicular to . This method requires that individual sections are acquired as volumes correctly reproducing their physical thickness. It is not possible to compensate for thickness variation that is the result of physical compression that differs between adjacent sections.

Boergens and Denk [7] developed an augmentation of the FIB-SEM to correct for section thickness variation during image acquisition. Later, they ascertain structural isotropy of the sample and transform -coordinates such that the peak of the curve of autocorrelations of several -cross-sections of the data has the same half width in both dimensions. Finally, the -coordinate of each section is determined by the average over all -cross-sections. This method enables automatic estimation of a scaling factor for sufficiently large section ranges that guarantees isotropy of the contained data assuming that local section thickness is constant.

Sporring et al. [8] operate on image data directly. Similar to our method, the position of sections is estimated by comparing a reference similarity decay curve with pairwise similarity estimates between sections. In contrast with our method, the reference curve is generated from in-section pixel intensities assuming strictly isotropic specimen. The context with which a decision is made is limited, since only the previous section in the series is taken into account for -position updates. Noise in individual sections is not considered.

Our method is unique among previous approaches in that it calculates an accurate globally consistent -position for each individual section directly from noisy post-acquisition image data without further demands on instrument design or imaging modality. It has the potential to calculate section thickness varying in and in a straight forward manner and does not require manual annotations or processing of section images. As a result, our method readily scales to large EM data sets being routinely collected.

3 Section Position Estimation

We start with the assumption that the spatial frequencies of tissue change slowly across the series of sections. As a result, the similarities between tissue sections shifted in space should decay with the magnitude of the shift. Since these tissue properties give rise to the image contrast in microscopy, we expect similarities between images of the tissue to reflect those properties. Our method corrects section spacing by estimating -shifts that match a locally smooth tissue similarity decay curve. In our experiments, we have used normalized cross correlation (NCC) as a measure for similarity. In addition, we are experimenting with other metrics that are less sensitive to correct section alignment such as the inverse false positive rate of invariant feature matches or the average correlation coefficient of local blockmatches. However, regardless of which similarity metric is employed, the inherent similarity decay curve is generally unknown. It therefore needs to be estimated from the data while simultaneously correcting -coordinates. Furthermore, each section is compromised by preparation artifacts (varying imaging conditions, differences in staining, etc.). Assuming that these quality differences apply to each section individually, the “quality” of a section affects all pairwise comparisons equally.

In Equation 1, we combine these assumptions into an optimization problem that jointly solves for

  1. a “tissue inherent” similarity decay curve,

  2. a quality measure for each section, and

  3. -shifts for each section such that observed similarity curves are best explained by the tissue inherent curve

(1)

In our notation, , and are indices for sections, is the value of the fit of the similarity localized at and evaluated at . and denote window functions that account for local smoothness,

is a vector of quality measures for each section,

holds the actual coordinates for each section, and holds the shifts with respect to the original (grid) positions. Finally, is the measured value of the similarity for reference at a distance .

This optimization problem has trivial solutions for all and can arbitrarily stretch and compress the series, both potentially leading to further trivial solutions. We therefore apply two bounding conditions:

  1. ,

  2. the minimum and maximum coordinates of the series remain unchanged, i.e. the series neither shifts nor scales globally.

As there is no closed-form solution to this non-convex optimization problem, we approximate Equation 1 by splitting it into three separate terms that we solve iteratively in alternating sequence until convergence is reached. Furthermore, an iterative approach gives us maximum flexibility to test a variety of further regularizing constraints. First, we estimate the similarity curve inherent to the tissue from the data (LF 1(a)). Second, we estimate the quality of each section (LF 1(b)) and, third, we calculate and apply the shift for each section (LF 1(b)). We repeat these steps until a specified number of iterations is reached.

Distance to reference section

Similarity

(a) Similarity curve estimate

Distance to reference section

Similarity

(b) Quality and shifts
Figure 2: Iterative optimization: (a) For each section we sample pairwise similarities at integral distances to the reference in transformed space and weight sample contribution by a windowing function centered at the reference section. is estimated by weighted averaging. We assume symmetry, i.e. 

. Function values at real valued coordinates are generated by linear interpolation. (b) All pairwise similarity estimates

at updated section positions are compared with generating a vote for both quality and position update.

4 experiments

We applied our method to two different data sets: (i) FIB-SEM,222FIB-SEM of Drosophila melanogaster CNS, courtesy of Ken Hayworth, Shan Xu, Harald Hess, HHMI Janelia dimensions 2048×128×1000 px, voxel-size 8×8×2 nm, and (ii) ssTEM,333ssTEM of Drosophila melanogaster CNS, courtesy of Rick Fetter, Zhihao Zheng, Davi Bock, HHMI Janelia dimensions 2580×3244×63 px, voxel-size 4×4×40 nm. The voxel depth of 2 nm and 40 nm, respectively, is the reported nominal section thickness. As we will see later, the actual thickness of individual sections varies greatly.

4.1 Data Set 1: FIB-SEM

The section thickness variation in the FIB-SEM image data is apparent from the -cross-section shown in LF 1. Tissue appearance changes abruptly from fast to slow at the indicated section. This indicates a sequence of thicker and thinner sections, respectively. Our method greatly reduces the severity of this artifact, as evidenced by the “squeezed” and “stretched” cross-sections through neuronal processes in the original image that appear round as expected after correction. We estimated a min and max section spacing of and  px (0.28 nm and 20.4 nm) respectively.

4.2 Data Set 2: ssTEM

In the ssTEM data set, the variances in section thickness are not as dramatic as in the FIB-SEM data. Therefore, we corrected thickness in the original data set and used the result as the basis for two revealing simulated data sets: (i) we removed sections from the original series to artificially create gaps within the series, and (ii) we randomly reordered sections in the series. All results are visualized in LF 3. We downsampled the data set by a factor of both in and for isotropic data.

As a numeric measure of quality, we calculate mean and max absolute difference between estimated -positions for the original stack and each modified stack. Any sections that have been removed during stack modification are ignored for this evaluation.

Figure 3: Results of the method applied to a TEM series and deliberate modifications. We show an -section before (original) and after correction (corrected), and the corresponding PSM with intensity-encoded NCC values in the same -coordinate frame. Arrows mark the places where sections were removed. White horizintal lines demark corresponding -positions across all -slices and PSMs for visual comparison. The estimated sections positions are almost identical in all three examples.

[style=unboxed,leftmargin=0cm]

Original:

The section thicknesses in the ssTEM data do not vary as much as in the FIB-SEM example. We estimated a min and max section spacing of and  px (24 nm and 64 nm) respectively. As a result, the corrected ssTEM -cross-sections do not appear dramatically different than the originals as is the case for FIB-SEM.

Missing Sections:

We removed sections . As a result, this stack contains five fewer sections than the original and structures are shifted compared to their original position (cf. LF 3). Our method correctly estimates a large distance between those sections adjacent to those that were removed. Due to the applied floor interpolation in LF 3, they are rendered as thick sections (see arrows). The estimated -positions of sections deviate from the corresponding -positions in the original reference series by 0.13 px (5.2 nm) on average, and 0.28 px (11.2 nm) at most.

Random Order:

We randomly reposition each section within in a range of and then apply our algorithm, allowing for the order of the sections to be changed whenever indicated by the respective -coordinates. Our method correctly recovers the initial order of the sections (LF 3). The estimated -positions of sections deviate from the corresponding -positions in the original reference series by 0.044 px (1.76 nm) on average, and 0.13 px (5.2 nm) at most.

4.3 Runtimes

We executed our experiments on a Dell Precision T7610 workstation and measured runtimes for our procedure. Similarity matrix estimation took  s and inference took  s for the FIB-SEM experiment with 1000 sections and 150 iterations. These steps took  s and  s respectively, for the TEM-experiments with 63 sections and 100 iterations.

5 Discussion & Future Work

We developed an efficient image based method to estimate accurate -positions of all images in microscopy image stacks with unknown varying -spacing. Our method can be used to estimate section thickness in complete section series. In addition, it can detect missing sections, however, it requires further inspection to distinguish gaps due to section loss from thick sections as the method reports only the position of individual sections. Finally, we demonstrated, that the method recovers the original section order in moderately randomized series which is particularly interesting for ssTEM where correct section order cannot be guaranteed.

In our experiments, we estimated a single -position for each section, however, we are planning to extend the method to estimate thickness variation within sections, employing a hierarchical coarse-to-fine approach.

In this paper, we tested our method on series of up to 1000 sections with substantial variance in section spacing and demonstrated its efficacy. Depending solely on one-time estimation of pairwise similarity between sections which can be achieved by sampling and in parallel, our method is not limited by the size of individual section images. In practice, we observed that sections that are 1000 or more pixels apart have very little impact on each other. Therefore, larger series can be split into independent overlapping batches that are processed in parallel and merged later. We believe that the correction of section position will improve the performance of both manual and automated segmentation methods by providing more consistent data across sections in , and will test this hypothesis in future work.

Our method is Open Source, publicly available in the ImageJ distribution Fiji [4], and integrated in TrakEM2 [3].

6 acknowledgements

This work was supported by HHMI. We thank Davi Bock, Ken Hayworth, Harald Hess, Rick Fetter, Shan Xu, and Zhihao Zheng for data collection and valuable discussion.

References

  • [1] G. J. Born, “Die Plattenmodellirmethode,” Archiv für Mikroskopische Anatomie, vol. 22, no. 1, pp. 584–599, 1883.
  • [2] K. L. Briggman and D. D. Bock, “Volume electron microscopy for neuronal circuit reconstruction,” Current Opinion in Neurobiology, vol. 22, no. 1, pp. 154–161, Feb. 2012.
  • [3] A. Cardona, S. Saalfeld, J. Schindelin, I. Arganda-Carreras, et al., “TrakEM2 software for neural circuit reconstruction,” PLoS ONE, vol. 7, no. 6, pp. e38011, June 2012.
  • [4] J. Schindelin, I. Arganda-Carreras, E. Frise, V. Kaynig, et al., “Fiji: an open-source platform for biological-image analysis,” Nature Methods, vol. 9, no. 7, pp. 676–682, July 2012.
  • [5] D. M. De Groot, “Comparison of methods for the estimation of the thickness of ultrathin tissue sections,” Journal of Microscopy, vol. 151, no. Pt 1, pp. 23–42, July 1988.
  • [6] M. L. Berlanga, S. Phan, E. A. Bushong, S. Wu, et al., “Three-dimensional reconstruction of serial mouse brain sections: Solution for flattening high-resolution large-scale mosaics,” Frontiers in Neuroanatomy, vol. 5, Mar. 2011.
  • [7] K. M. Boergens and W. Denk, “Controlling FIB-SBEM slice thickness by monitoring the transmitted ion beam,” Journal of Microscopy, vol. 252, no. 3, pp. 258–262, Dec. 2013.
  • [8] Jon Sporring, Mahdieh Khanmohammadi, Sune Darkner, Nicoletta Nava, et al., “Estimating the thickness of ultra thin sections for electron microscopy by image statistics,” in 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), Apr. 2014, pp. 157–160.