Fully Automatic 3D Reconstruction of Histological Images

07/18/2009 ∙ by Ulas Bagci, et al. ∙ IEEE 0

In this paper, we propose a computational framework for 3D volume reconstruction from 2D histological slices using registration algorithms in feature space. To improve the quality of reconstructed 3D volume, first, intensity variations in images are corrected by an intensity standardization process which maps image intensity scale to a standard scale where similar intensities correspond to similar tissues. Second, a subvolume approach is proposed for 3D reconstruction by dividing standardized slices into groups. Third, in order to improve the quality of the reconstruction process, an automatic best reference slice selection algorithm is developed based on an iterative assessment of image entropy and mean square error of the registration process. Finally, we demonstrate that the choice of the reference slice has a significant impact on registration quality and subsequent 3D reconstruction.



There are no comments yet.


page 6

page 7

page 15

page 16

page 17

page 18

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

2D imaging methods, such as optical microscopy, are still preferable to 3D imaging methods due to their high level of specificity and high resolution properties. Histological sections (slices) obtained through 2D imaging methods provide useful information for the diagnosis or the study of pathology. Although 2D histological slices have great impacts on quantification and visualization of clinical data, 3D volume reconstruction from these 2D slices is required in order to fully appreciate anatomical structures [1].

Typically, a 3D volume is reconstructed by registering (aligning) the 2D sections with respect to a chosen reference and stacking successive aligned sections [2]. As the acquisition processes of different 2D histological images are performed independently, slice misalignment and deformation is often unavoidable. The deformation varies from section to section and non-cohorent distortions may exist in consecutive sections. Choosing an arbitrary slice as a reference slice leads to errors in 3D volume reconstruction, hence, the reference slice should be chosen properly not to contain distortions in order to achive high quality volume reconstruction [3, 4].

Fully automatic registration of histological slices and reconstruction of 3D volume are necessary for two reasons. First, since manual registration using interactive alignment is non-reproducible and user dependent, it cannot be used if the number of slices is large [1, 2]. To quantify changes between images, motion and deformation characteristics specify the type of transformation (registration). Since histological slices change smoothly from slice to slice and the section distortions induced by the preparation process are local in nature [3, 5, 4], accurate alignment of these slices can be achived by using elastic registration methods. Second, since manual selection of the best reference slice (BRS) uses qualitative measures and ignores the image information content, optimum smooth 3D volume reconstruction cannot be guaranteed.

In this paper, we present a fully automatic 3D reconstruction method which tackles three difficult problems in registration of histological images. Section 2 explains an important preprocessing method, called standardization, which captures intensity variations between slices and plays a significant role in identifying BRS selection. In Section 3, the edgeness space is presented for the registration framework to provide better global alignment and to avoid possible misalignments. Section 4 briefly explains locally affine globally smooth (LAGS) registration method. Based on iterative assessment of image entropy and Mean Square Error (MSE) of the registration process in feature space, an automatic BRS selection algorithm is described in Section 5

. To evaluate reconstructed volume qualitatively and quantitatively, we use both Correlation Alignment Measure (CAM) and proposed Standard Deviation Maps (SDM) in Section 

6. Evaluations and discussions are given in Section 7.

2. Standardization of Image Intensity Scale

Image intensity variations are not only influenced by the distribution of light sources, but also the content (different tissues) of the images as different tissues show different intensity levels. To avoid the effects of illumination conditions and identify those intensity variations due to different tissue types, a standardization procedure is applied to histological images.

Standardization is a non-linear pre-processing technique which maps image intensity histogram (scale) into a standard intensity histogram (scale) so that similar intensities will have similar tissue meaning after standardization. Standardization was firstly developed for MR images [9]. In previous works [3, 4], we applied this method to standardize histological images.

3. Feature Space

Choice of the feature space plays a significant role in image registration especially if the similarity metric is based on the optimization function independent of spatial information such as mutual information. Since these kind of registration methods do not take into consideration the spatial information of pixel/voxel intensity distribution/variation, the optimization algorithm may get stuck in local maximum resulting in misalignment. Defining a feature space capturing variations of gray-level characteristics will overcome the drawbacks of intensity based approaches. To align the images globally, we used a particular feature space which represents an image by continuous variables, called edgeness, and describes the intensity variance of a predefined region over the image 

[10, 3, 4].

We represent an image (section/slice/scene) by a pair where is a two-dimensional (2-D) array of scene elements (pixels) and is intensity function, whose domain is . We assign an integer intensity value for each pixel . Edgeness feature space is defined by the pair , where is a fixed radius for each region. At image coordinate , the edgeness is represented by


where is the radius. It should not be concluded that the process is just emphasizing edges and deciding whether a specific voxel/pixel belongs to the edge or not [10, 3, 4]. Instead, within a specified radius value, the image feature content is forced to stay beyond a variation level which prevents the registration process from getting stuck in local maxima.

4. Locally Affine Nonlinear Transformation

Local alignment (elastic) of images is obtained through Locally Affine Globall Smooth (LAGS) registration method described in [7, 8, 3]. Since consecutive slices are not exactly the same, rather slices vary smoothly, LAGS registration algorithm fits well to the problem. For 2D images, 8 affine parameters are needed to fully identify changes between images. Two of these affine parameters are needed to capture local brightness and contrast patterns [7] and 6 affine parameters are used to capture local deformations for 2D images. Since the standardization procedure has been used to remove intensity variations among the same tissue types, there is no need to use these 2 affine parameters. Briefly, LAGS registration algorithm uses the difference image of source and target image as an optimization function and tries to minimize it over small local image domain. Readers are strongly encouraged to read [7, 8, 3, 4] to understand the theory of LAGS and the modified algorithm which takes into account the standardization procedure.

5. Automatic Best Reference Slice Selection

The quality of the 3D volume reconstruction process mostly depends on the choice of the reference slice. The reference slice is used as a target image and all the remaining slices are being considered as source images to be registered onto the target image. If the reference slice is distorted or noisy, reconstructed 3D volume will not be optimal. Once the reference slice is identified as target image, registration based fusion methodology can be applied for reconstruction [2].

Selecting best reference slice can be based on high confidence image features such as MSE, entropy, edge, texture, color, intensity histograms, etc.

  1. MSE:

    In the case of distortions, structural discontinuity is not minimum even for the consecutive slices. When affine registration is performed for global alignment of images, the optimization procedure tries to minimize MSE between images but due to distortion, it will not reach low MSE values. Furthermore, it is also known that with small SNR values, alignment is difficult, leading to high registration errors. Therefore, MSE can be used as a tool for checking whether the slices are distorted or not. While high MSE values indicate most probably distorted and noisy slices, low MSE values indicate strong similarity between consecutive images.

  2. Edge: In feature space, we emphasise edgeness features of an image by mapping image space into the feature space where edgeness parameters hold both edge information and spatial variations of pixel intensities over all regions in the image. Therefore, we assume that MSE between any image pair already includes high confidence information related to edges.

  3. Contrast/Brightness: Contrast/brightness patterns also play an important role in image contents. Since the standardization method has been used to correct intensity variations, intensity for the same tissue is the same for all images.

  4. Entropy: Entropy is another measure often used to characterise the information content of a data source. It has been used as a metric for image registration in the form of mutual information. Large mutual information between images implies high similarity and vice versa.

To select BRS automatically, one needs to define a metric which describes the reference slice in terms of noise, distortion and information content levels. As in [3, 4], BRS can be formulated as


where is the feature space based mean square error after registering the image into the image and is entropy. Further theoretical details of BRS selection algorithm can be found in [3, 4].

6. Implementation, Experiments and Results

Registration of histological slices requires serial registration procedure which is just a combination of transformation functions. Let be the transformation function that warps the source image into the target image . The transformation is computed serially as follows


where represents the composition.

One of the advantages of using the subvolume approach is to avoid the ”banana shape” effect resulting from the reconstuction process [1, 2]. In serial registration, one of the disadvantages of using affine registration is that we lose the topology of the reconstructed volume. To reduce or eliminate this effect, one may need to use MRI of the volume superimposed onto the reconstructed histological volume or use rigid registration.

In summary, registration is performed initially for slices in each subvolume separately. Three kinds of registration are performed in the reconstruction process: rigid, affine and LAGS. MSEs are calculated according to affine registration in edgeness space and have been used to select BRSs for each subvolume. Affine registration is performed in a serial manner combining transformation functions. Then, LAGS registration is performed to capture local deformations in each subvolume with respect to the chosen reference. Once LAGS registration has been finished, subvolumes are registered to each other in a rigid manner.

6.1. Evaluations and Results

We have registered a stack of 350 Nissl-stained slices acquired by cyro-sectioning coronally on an adult mouse brain with a resolution of 590x520 pixels at a resolution of 15m and 24-bit color format [11].

Quantitative evaluation of the results of the reconstruction process is often difficult. It has been shown in [12] that an ideal measure of the quality of the reconstruction is the smoothness of the reconstructed surfaces. In this work, they propose a new measure based on evaluation of smoothness of the reconstructed volume called Correspondence Alignment Measure (CAM). As an alternative method to CAM, Standard Deviation Maps (SDM) is proposed to measure the smoothness of the reconstructed volume. The method is based on the standard deviation of the pixel values for the same location in each section. The CAM and SDM results for the reconstructed 3D volume and validation of the SDM are given in the following subsections.

6.1.1. Cam

The CAM measure relies on the assumption that if a point is perfectly aligned, it lies midway between its corresponding points on neighbors’ sections. To compute the CAM measure for a given image, first of all, corresponding points for specifed control points in the image are identified. The associated confidence values in two adjacent images are then calculated. If the confidence is greater than a pre-defined threshold

, square root of the summation of the deformation vectors are added to the cumulative sum. Finally, the cumulative sum is normalized by the number of pixels which have contributed. Note that CAM gives one value for each image, therefore, mean or standard deviation of CAM values of serial images are needed to compare reconstructions. Reconstucted volume is smooth if the mean or the standard deviation of CAM measures are low and vice versa.

Summary of the changes in mean and standard deviation in CAM values is given by Table 1. The values in Table 1 are obtained by considering the worst case which uses all the slices instead of just a few slices from the middle of the stack as defined in [12], and is set to . Even for the worst case, CAM values indicate that a smooth volume is constructed with the proposed framework. While mean values dropped by % and %, the standard deviation values dropped by % and % for affine and locally affine registered stacks respectively.

Figure 1. CAM for original and registered slices
Rigid Reg. Affine Reg. LAGS Reg.
Mean 55.911 51.832 45.461
Std 12.223 9.232 8.833
Table 1. CAM-mean and standard deviation values for reconstructed 3D

6.1.2. Sdm

We offer here a simple way of measuring the quality of the reconstruction by considering the smoothness of the reconstructed surfaces. If the reconstructed volume is naturally smooth, it means that the structures change smoothly and slowly from slice to slice which highly depends on the registration quality.

Assumption: For any reconstucted volume , which is a sequence of images , , if we take the same pixel position for all slices in , one should expect smooth transition of pixel values within the slices of , if the slices are well registered.

To validate this assumption experimentally, a 3D MRI volume is taken and its slices are warped by applying random deformations spanning high-to-low level. The randomly warped slices are used to reconstruct a warped 3D MRI volume for which we compute SDM and compare it with the smoothness of .

Figure 2 shows the experimental validation of SDM for different level of deformations applied to the slices of . While the first SDM (the first image in the first row) has carried the highest deformation level, the level of deformation has been decreased until the last SDM (the third image in the second row) is obtained, which carries no deformation level, itself. As the deformation level decreases, the smoothness level of approaches the original volume .

Figure 2. SDMs for warped volumes spanning from high-to-low level of deformations

SDMs for the reconstructed volume for rigid, affine and LAGS are shown in Figure 3, both in gray scale and spectrum format. Although the volume reconstructed by successive affine registrations is smoother than the volume reconstructed by successive rigid registrations, it includes the ”banana-shape” effect which can be corrected by superimposing MRI of the rat brain (if available) on the reconstructed volume. Among three methods, the smoothest reconstructed volume is obtained by the proposed method shown in the last SDM in Figure 3.

Figure 3. Standard Deviation Maps for Rigid registered, Affine-Registered and Elastic-Registered Stacks

7. Conclusion

In this paper, we have presented a novel framework to reconstruct 3D rat brain volume from 2D histological images. The framework is based on three fundamental premises. (1) All histological images must be standardized for accurate registration leading to 3D volume reconstruction. (2) For accurate and succesful registrations in consecutive slices, a reliable feature space must be taken into account. (3) For automatic 3D volume reconstruction, the reference slice must be chosen properly by avoiding slices with high noise, distortions and other factors. To validate the reconstructed volume, the smoothness of the volume is considered. In addition to the existing method CAM, we have proposed a method called SDM to measure the smoothness of the reconstructed volume. Qualitative and quantitave evaluation of experimental results indicate that the reconstructed volume is highly accurate.

8. Acknowledgements

We wish to thank Prof. Jayaram K. Udupa, from Medical Image Processing Group of University of Pennsylvania, for his encouragement and guidance throughout this project.


  • [1] S. Ourselin, A. Roche, G. Subsol, X. Pennec, and N. Ayache, “Reconstructing a 3d structure from serial histological sections,” Image and Vision Computing 19, pp. 25–31, September 2000.
  • [2]

    G. Malandain, E. Bardinet, K. Nelissen, and W. Vanduffel, “Fusion of autoradiographs with an MR volume using 2-D and 3-D linear transformations,”

    NeuroImage 23, pp. 111–127, September 2004.
  • [3] U. Bağcı, “Registration of standardized histological images in feature space” SPIE Medical Imaging, Conference on Image Processing 2008, To Appear.
  • [4] U. Bağcı, “Fundamental issues of registration: Applications for change analysis in health and disease,” CMIAG Annual Report , 2007.
  • [5] T. Ju, J. Warren, J. Carson, M. Bello, I. Kakadiaris, W. Chiu, C. Thaller, and G. Eichele, “3d volume reconstruction of a mouse brain from histological sections using warp filtering,” J. Neurosci. Methods (156), pp. 84–110, 2006.
  • [6] J. C. Gee, D. Haynor, L. Briquer, and R. Bajcsy, “Advances in elastic matching theory and its implementation,” in CVRMed, pp. 63–72, 1997.
  • [7] S. Periaswamy and H. Farid, “Elastic registration in the presence of intensity variations,” IEEE Transactions on Medical Imaging 22, pp. 865–874, July 2003.
  • [8] U. Bağcı and L. Bai, “Multiresolution elastic medical image registration in standard intenstiy scale,” in 20th Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI), October 2007.
  • [9] Y. Ge, J. Udupa, L. Nyul, L. Wei, and R. Grossman, “Numerical tissue characterization in ms via standardization of the mr image intensity scale,” Journal of Magnetic Resonance Imaging 12, pp. 715–721, November 2000.
  • [10] T. Butz and J.-P. Thiran, “Affine registration with feature space mutual information,” in MICCAI ’01, pp. 549–556, (London, UK), 2001.
  • [11] A. MacKenzie-Graham, E. Lee, I. Dinov, M. Bota, D. Shattuck, S. Ruffins, H. Yuan, F. Konstantinidis, A. Pitiot, Y. Ding, G. Hu, R. Jacobs, and A. Toga, “A multimodal, multidimensional atlas of the c57bl/ 6j mouse brain,” J.Anat (204), pp. 93–102, 2004.
  • [12] E. Guest and R. Baldock, “Automatic reconstruction of serial sections using the finite element method,” BioImaging 3, pp. 154–167, May 2001.