A Deep Learning Approach for Semantic Segmentation of Unbalanced Data in Electron Tomography of Catalytic Materials

01/18/2022
by   Arda Genc, et al.
PNNL
The Ohio State University
0

Heterogeneous catalysts possess complex surface and bulk structures, relatively poor intrinsic contrast, and often a sparse distribution of the catalytic nanoparticles (NPs), posing a significant challenge for image segmentation, including the current state-of-the-art deep learning methods. To tackle this problem, we apply a deep learning-based approach for the multi-class semantic segmentation of a γ-Alumina/Pt catalytic material in a class imbalance situation. Specifically, we used the weighted focal loss as a loss function and attached it to the U-Net's fully convolutional network architecture. We assessed the accuracy of our results using Dice similarity coefficient (DSC), recall, precision, and Hausdorff distance (HD) metrics on the overlap between the ground-truth and predicted segmentations. Our adopted U-Net model with the weighted focal loss function achieved an average DSC score of 0.96 ± 0.003 in the γ-Alumina support material and 0.84 ± 0.03 in the Pt NPs segmentation tasks. We report an average boundary-overlap error of less than 2 nm at the 90th percentile of HD for γ-Alumina and Pt NPs segmentations. The complex surface morphology of the γ-Alumina and its relation to the Pt NPs were visualized in 3D by the deep learning-assisted automatic segmentation of a large data set of high-angle annular dark-field (HAADF) scanning transmission electron microscopy (STEM) tomography reconstructions.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 6

page 8

page 9

07/11/2017

Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations

Deep-learning has proved in recent years to be a powerful tool for image...
04/05/2019

Multiphase Level-Set Loss for Semi-Supervised and Unsupervised Segmentation with Deep Learning

Recent state-of-the-art image segmentation algorithms are mostly based o...
02/19/2019

Accurate Automatic Segmentation of Amygdala Subnuclei and Modeling of Uncertainty via Bayesian Fully Convolutional Neural Network

Recent advances in deep learning have improved the segmentation accuracy...
05/22/2019

WPU-Net:Boundary learning by using weighted propagation in convolution network

Deep learning has driven great progress in natural and biological image ...
01/18/2018

On the influence of Dice loss function in multi-class organ segmentation of abdominal CT using 3D fully convolutional networks

Deep learning-based methods achieved impressive results for the segmenta...
02/17/2022

Effective Training Strategies for Deep-learning-based Precipitation Nowcasting and Estimation

Deep learning has been successfully applied to precipitation nowcasting....
01/10/2022

3D Segmentation with Fully Trainable Gabor Kernels and Pearson's Correlation Coefficient

The convolutional layer and loss function are two fundamental components...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

TEM tomography in materials science has become the de facto technique that enables valuable high spatial resolution information on the structure of the materials in 3D [3, 31, 30, 32, 48]. Despite the progress in developing novel methods in acquiring and aligning the tomography tilt series and a broad spectrum of reconstruction algorithms[13, 47, 14, 41], semantic segmentation of the large 3D data sets remains a significant bottleneck in 3D analysis. Manual segmentation is a very time-consuming task, relying heavily on the handcrafting skills and expertise of a human operator. A reliable, reproducible, and fully automated segmentation method is in high demand for scaling the 3D data analysis and collecting statically meaningful information where a single reconstructed volume segmentation involves hundreds of images or more.

Recent advances in deep learning methods have revolutionized the field of computer vision

[24, 22, 26, 4], and the evolution of these methods has enabled the automatic semantic segmentation of large data sets; otherwise, manual analysis is unfeasible[33, 5, 6]

. Deep learning pixel-wise classifiers have been successfully applied to many semantic segmentation tasks where complex structures are not easily mapped by simple intensity differences, and boundaries between the image features are not apparent due to the variations in contrast gradients

[10].

In deep learning, fully convolutional neural networks (FCNs) hierarchically recognize complex features directly from the training data without the additional feature engineering. More recently, FCNs, inspired by large and deep networks, are efficiently trained end-to-end by supervised learning and pixel-to-pixel probabilities computed successfully, thanks to the many advancements in parallel computing

[9]. It has been shown that segmentation results using FCNs can indeed reach a human-level performance and even on some occasions exceed that without the post-tuning of the results[50, 11, 38]

. Today, it is possible to generalize these deep networks with a limited amount of ground-truth data by implementing data augmentation and regularization techniques while mitigating the problem of variance

[16].

In this paper, we explore the capability of FCNs in the semantic segmentation of a -Alumina ()/Pt catalytic material. Historically, -Alumina has been one of the most used catalytic support materials for noble metals and oxide catalysts employed for reduction, oxidation, and reforming reactions in automotive exhaust control and petroleum refining processes[23]. -Alumina possesses a complex crystalline structure; despite its well-known application space, the origin of the catalytic behavior is not well understood[21, 20]. There is considerable debate on the role of surfaces of -Alumina responsible for both catalytic behavior and anchoring of the noble metallic NPs. In addition to the structural complexity and small crystallite sizes, the support material -Alumina consists of a dense network of matrix pores, and the degree to which these pores are connected to outside surfaces is of great interest[21, 20, 39, 8].

In catalytic materials, the sparse distribution of the noble metallic NPs, over the background and oxide support material introduces an unbalanced representation of the data in TEM images. The extend of the class imbalance problem between the foreground and background of the images has been extensively studied in deep learning-based semantic segmentation approaches[36, 34, 43]. It has been shown that the choice of loss function significantly impacts the performance of a semantic segmentation model[49]. Many recent state-of-the-art applications of FCNs focus on the implementation of weighting strategies coupled with distribution-based or differentiable region-based loss functions for the optimization of the models[49, 19, 42].

Even though weighting strategies at the loss function level control the class imbalance, the problem of loss becoming overwhelmed by the number of easy examples during inference remains a challenge in complex multi-class situations[25]. Moreover, there are limited applications of these strategies in the semantic segmentation of materials science samples, particularly segmentation of the 3D electron tomography reconstructions[5, 38, 17, 37].

We present a U-net-based FCN architecture and weighted focal loss as a loss function to overcome the class imbalance problem[40, 25]. The weighted focal loss is a distribution-based loss function, and weighting of the unbalanced data occurs at the loss function level in contrast to data preprocessing strategies. In addition to the weighting, focal loss applies a modulation term to the standard cross-entropy loss and dynamically scales the confidence of the correctly classified examples[25]. In our experiments, the U-Net architecture equipped with the weighted focal loss facilitated a comprehensive 3D representation of the catalytic material and provided a clear insight regarding the long-standing debate on the characteristics of -Alumina surfaces and their relation to the catalytic NPs. We discuss the accuracy of our segmentation results by assessing commonly used semantic segmentation metrics on the overall overlap and boundary match between the ground-truth and predicted segmentations.

To further test our model’s robustness and validity, the best-performing model was applied to the automatic semantic segmentation of a large data set of reconstructed images. We believe strongly that deep learning-based semantic segmentation methods have immense potential in 3D data analysis and will usher in a new era in materials design and discovery.

2 Methods

2.1 3D tomography data acquisition and visualization

Our main goal is to assess the effectiveness of a deep learning-based approach in semantic segmentation of the 3D HAADF STEM tomography reconstructions while achieving a full 3D view of the -Alumina/Pt catalytic material. For this purpose, we conducted HAADF STEM tomography experiments on a well-isolated -Alumina/Pt catalytic particle. TEM samples were prepared by dropping a solution containing well-dispersed NPs on a lacey carbon film. A detailed description of the synthesis of -Alumina/Pt material was reported in an earlier paper[21]. A probe aberration-corrected 300 kV Thermo Fisher Scientific Titan S/TEM microscope was used for the acquisition of the HAADF STEM tilt series. HAADF STEM images were acquired at the detector inner collection angle of 40 mrad, beam current of 20 pA with a 0.1 nm probe size, an accelerating voltage of 200 kV.

To extend the depth of focus of the electron beam during STEM tomography acquisitions, the convergence angle of the illumination system was adjusted to 10 mrad using the three-condenser lens optics of the microscope. Tomography tilt series consists of 69 HAADF STEM images acquired at the tilt range of 68 and tilt increment of 2.

Post-processing of the tilt series was conducted using open-source resources; a Python programming script based on the tomoviz software used for the image shift and tilt alignments

[46]

, and TomoPy and ASTRA Toolbox Python libraries for the maximum likelihood expectation maximization (MLEM) reconstructions

[35, 12]. Paraview software was employed to generate 3D visualizations from the fully segmented 3D reconstructions[2].

2.2 Segmentation architecture

For FCNs experiments, we exploit an adopted version of the U-Net architecture, and a schematic of the architecture is shown in Figure 1. Our network consists of two learning paths, a down-sampling (contraction) path and an up-sampling (expansion) path. There are six convolutional steps in the down-sampling path and five in the up-sampling path. In the down-sampling path, each step has two convolutional layers with a filter size of 3x3. The size of the feature maps is halved by the pooling layers following each step. In the up-sampling path, each step starts with a convolutional transpose layer with a filter size of 2x2 and a stride of 2, followed by two convolutional layers with a filter size of 3x3. The size of the feature maps is doubled, and the number of feature maps is halved at the end of each convolutional step in the up-sampling path.

In our version of the U-Net, we used the ‘same’ padding in the convolution layers followed by an average pooling layer for down-sampling. Using the ‘same’ padding resulted in the output of the convolution layers being the same size as the input layers. The pooling operation plays a vital role in the flow of information through the convolutional layers and defines the model’s sensitivity to details. In our architecture, average pooling is preferred over max pooling for down-sampling to reduce the spatial information loss at the feature boundaries and prevent excessive pixel saturation.

Concatenation paths (skip connections) give the network a well-known ‘U’ shape pattern and link the high spatial information from down-sampling convolutional layers to the up-sampling convolutional layers. The network hierarchically learns the contextual information and fine details in the predicted images. We did not see a significant performance improvement in our model with the addition of dropout layers; instead, we employed a comprehensive data augmentation strategy for regularization. We used a rectified linear unit (ReLU) activation function for hidden layers and softmax for the output convolutional layer with final feature maps of 3.

2.3 Training and optimization

For optimization, we used a mini-batch gradient descent with a batch size of 2 and Adam optimizer at a learning rate of 0.0005. We used default parameters from the Tensorflow deep learning framework for the first and second moments of gradient averaging and updating

[1]. All the weights were initialized by ”He normal” kernel initialization[15], and all the biases were initialized at 0. A total of 30 ground-truth images were selected from the 3D reconstructions, and corresponding ground-truth segmentations were manually annotated for the training and validation steps. A class label representing background/pores, -Alumina, and Pt NPs, were assigned to each pixel in the ground-truth segmentations.

Due to the large image size and limited GPU memory availability, 1024x512 pixels (0.12 nm/pixel) ground-truth images and segmentations were divided into 512x512 pixels patches. This data was then randomly split into 75% training and 25% validation data sets. The average pixel density of each class in the patches is 77.7% for background/pores, 21.8% for -Alumina, and 0.5% for Pt NPs.

Data augmentation is crucial to teach the network a robust invariance to input data and generalize the model. We used rotation, vertical and horizontal flip, zoom, and shear transformations during training to generate a diverse range of images representing variations in the location and shape of the features. The best-performing model was selected based on the evaluation performance and applied to the automatic segmentation of a stack of 702 3D reconstructions. We employed a smooth blending approach to form final predictions where 512x512 pixels size segmented patches were smoothly merged into 1024x512 pixels size final predictions using spline interpolations between the overlapping patches

[28].

Figure 1: A schematic of the U-Net architecture. The number of feature maps is indicated on the top of each box, and the dimensions of the feature maps are on the bottom left corner. Orange boxes show the contraction path, and black boxes show the expansion path. The gray arrows and boxes indicate the concatenation path at each convolutional step.

As described previously, we investigate the complex bulk and surface structure of the -Alumina/Pt catalysts. Our segmentation model aims to distinguish the two-phase microstructure of the -Alumina and Pt NPs, as well as the pores. A correct classification of the surfaces, defined by the -Alumina - background and -Alumina – Pt - background boundaries, is mainly of high interest. In the tomography reconstructions, -Alumina and the background constitute the most significant fraction of the reconstructions, compared with the sparsely distributed nanoscale Pt particles. When there is an imbalance in the data representation, the learning algorithm can be biased towards the dominant class, represented densely by the higher number of pixels. In order to address this problem, we used weighted focal loss instead of the standard cross-entropy to minimize the loss[25, 29].

Weighted focal loss is a differentiable modification of the cross-entropy loss term and addresses the class imbalance problem in two ways. Firstly, it shifts the focus from easy-to-classify pixels towards hard-to-classify pixels by extending the range in which each pixel receives loss. This is achieved by scaling the cross-entropy loss by a focusing parameter and preventing the loss from being overwhelmed by the easy pixels. Secondly, the role of the weighted focal loss is to provide a balancing action on the class imbalance problem by adjusting the contribution of each class to the loss function by a weighting factor. Categorical focal loss () is defined as the following in a multi-class problem:

(1)

where and are the ground-truth and prediction probabilities of class c at pixel location i . Parameters C and N are the number of classes and pixels, respectively. is the weighting factor for class c and

is the focusing parameter. Both focusing parameter and weighting factor are tunable hyperparameters. In our experiments,

values were approximated on the density of representation of each class at the range from 0 to 1, and was set to 1.

2.4 Evaluation metrics

Evaluation metrics play an essential role in proving the network’s performance and thus establishing the model for automatic semantic segmentation. In this work, prediction results were assessed using four commonly used semantic segmentation metrics: Dice similarity coefficient (DSC), recall, precision, and Hausdorff distance (HD)[45]

. DSC, recall, and precision scores are similarly extracted from the confusion matrix and defined as:

(2)
(3)
(4)

where true positives (TP), false positives (FP), and false negatives (FN) represent per pixel classifications of the confusion matrix.

We also evaluated our semantic segmentation results by measuring the dissimilarities specifically at the segmentation boundaries. Hausdorff distance is a boundary distance-based metric and measures the largest segmentation error in the overlap between the ground-truth and predicted segmentations[18]. Given two sets of points A and B, Hausdorff distance is defined as:

(5)

where hd(A,B) and hd(B,A) are directed Hausdorff distances:

(6)
(7)

Functions hd(A,B) and hd(B,A) measure the distances between two points in A and B, which are farthest from any nearest neighbors, and HD (A,B) gives the largest of these distances. ‖a-b‖ is the Euclidean norm between the points in A and B. A well-documented behavior of the Hausdorff distance is its sensitivity to outliers and noise

[27]; thus, we report robust HD (RHD) values considering the percentile of the largest segmentation errors and as well as the maximum HD[7, 44]. We aim to down-weight the impact of outliers and noise on the HD metric by measuring the RHD values.

3 Results and discussion

Segmentation of the HAADF STEM tomography reconstructions is a challenging task due to information loss from the insufficient number of projections (i.e., missing wedge artifacts) and variations in the contrast and size of the features in tilt images. A series of representative orthogonal slices (orthoslices) taken from the reconstructed 3D volume of an isolated -Alumina/Pt particle is shown in Figure 2. a-c.

Figure 2: Orthogonal slices taken from the 3D HAADF STEM reconstructions of an isolated -Alumina/Pt particle.
Figure 3: A set of ground-truth and predicted segmentations of -Alumina and Pt NPs from the validation data and corresponding false negative and false positive maps showing the discrepancies in the overlap of each class.

In the orthoslices, the 3D microstructure of the particle is sectioned perpendicular and parallel to the broad surfaces of the -Alumina. At first glance, we notice the significant contrast difference between the oxide -Alumina and metallic Pt NPs, where Pt NPs appear much brighter than both -Alumina substrate and background in the reconstructions. Pt NPs mainly exhibit round shapes, and their size distribution is in the 1- 4 nm range, while -Alumina particles have a thin plate-like structure and a roughly rhombus shape. Some of the very small Pt NPs show an elongated shape because of the well-documented missing wedge artifact in TEM tilt tomography[31], as seen in Figure 2.c. In -Alumina, the crystallographic shape of the particle is defined by the {110} and {111} type surfaces, and the main ‘broad’ surface is {110} orientation, and side surfaces are {111}. Furthermore, synthesized -Alumina particle shows small ledges and facets on the surfaces and a network of high-density pores inside the matrix.

To evaluate the performance of our model in the segmentation task, we compared the results obtained from the validation data. Figure 3. visualizes results of exemplar qualitative segmentations from -Alumina and Pt NPs, respectively. The images represent two 512x512pixels patches from the validation data. For each patch of the validation data, ground-truth and predicted segmentations of -Alumina and Pt NPs are compared separately in the binary images. The differences in the segmentations are shown more explicitly in the false negative and false positive maps highlighting the discrepancies in the overlap of each class. When considering the false positive maps of the -Alumina segmentations, we notice that -Alumina boundaries are relatively shifted towards the pores inside the -Alumina matrix. In contrast to -Alumina/pore boundaries, most misclassifications near the surfaces are discontinuous. False negative and false positive regions are extended only a few pixels wide towards either -Alumina or background. Moreover, the boundaries in the predicted segmentations, both on the surfaces and along the -Alumina/pore boundaries, are smoother than the ground-truth segmentations.

We further discuss these observations in the context of the evaluation metrics, and the results are shown in Table 1. As expected, shifting of the -Alumina boundaries, mainly towards the pores, is reflected in the evaluation results such that the average precision score is lower than the average recall score for

-Alumina. The average precision and recall scores are 0.95

0.008 (mean

95 confidence interval) and 0.97

0.004, respectively.

In contrast to -Alumina, we observed fewer false positives in the segmentations of Pt NPs. Still, there are some missing Pt NPs in the predictions and corresponding misclassified regions seen in the comparison of the Pt NPs segmentations. Compared with -Alumina, the average precision score of Pt NPs is higher than the recall score. The average precision and recall scores are 0.92 0.03 and 0.78 0.04, respectively for the segmentation of Pt NPs.

One explanation for the extension of the -Alumina matrix towards the pores inside may be the contrast modulations along the diffuse boundaries between the -Alumina and pores. Uncertainty in the contrast of these boundaries can potentially fuel ambiguity in the manual annotations of the ground-truth segmentations. Still, a precise annotation of these low contrast boundaries can be a challenging task even for a human expert. Another approach would be generating more annotated data to elevate the model’s performance. However, this would be computationally costly and would require additional manual annotations. It is also worth mentioning that it is inevitable to have false negatives and false positives during inference. Practically, we would aim a trade-off between precision and recall.

Nevertheless, a comparison of the ground-truth segmentations with the predicted segmentations shows a strong correlation, especially in the complex surface structure of the -Alumina and the appearance of the Pt NPs. The overall similarities in the size and shape of the Pt NPs between the ground-truth and predicted segmentations also suggest that the model has convincingly managed the class imbalance problem without a significant underestimation. The overall segmentation performance measured by the DSC score for each class is 0.99 0.002 for background/pores, 0.96 0.003 for -Alumina, and 0.84 0.03 for Pt NPs. Here, we report the background DSC score since the pores inside -Alumina are associated with the background class.

Evaluation Metrics -Alumina Pt NPs Background\Pores
Precision 0.95 0.008 0.92 0.03 0.99 0.001
Recall 0.97 0.004 0.78 0.04 0.99 0.003
DSC 0.96 0.003 0.84 0.03 0.99 0.002
Table 1: Evaluation results from the validation data. All the values are in the form of mean %95 confidence interval.
Figure 4: A plot of robust Hausdorff distance (RHD) vs. percentile of the largest segmentation error. Data points are in the form of mean standard error.

To investigate our segmentation results further, we conducted measurements on the degree of boundary match using the HD metric. This is of particular interest for detecting the model’s performance, correctly learning the boundaries in each class. The HD metric is a powerful tool in measuring the largest segmentation error in the overlap between the two segmentations, while the DSC score is an overlap metric affected by the segmentation performance over the entire image. Figure 4. shows the trend in the average HD of -Alumina and Pt NPs segmentations at the various percentiles of robust HD (RHD) values. Measurement of the RHD provides a unique opportunity to understand the contributions of the outliers and noise to the model performance while guiding the degree of boundary match. As seen in Figure 4., RHD values fall sharply from a maximum HD of 5.45 1.78 nm (mean standard error) for -Alumina and 3.70 1.84 nm for Pt NPs. At RHD95, HD between the ground-truth and predicted segmentations decreases to 1.45 0.68 nm for -Alumina and 3.48 1.80 nm for Pt NPs, and at RHD90, the largest segmentation errors are less than 2 nm (0.59 0.05 nm for -Alumina and 1.81 1.05 nm for Pt NPs). The variations in the RHD values show that the largest segmentation error for the Pt NPs trend higher than for the -Alumina, which is consistent with the overall lower evaluation scores observed for the Pt NPs, particularly the lower average recall score. Yet, this analysis suggests that our model learned reasonably well to classify the pixels at the boundaries, especially those of the -Alumina in addition to the matrix regions.

Figure 5: Comparison of the test images (3D reconstructions) and predicted segmentations of -Alumina catalytic support material and Pt NPs.

Based on the evaluation results, we applied the best-performing model for the automatic semantic segmentation of a large volume of 3D reconstructions. Figure 5. compares the example reconstructions extracted at various locations along the 3D volume of the -Alumina/Pt particle and corresponding predicted segmentations by the model. These reconstructions are test images, and the model has not seen them during the training and validation steps. In the predicted segmentations, the pixels classified as -Alumina are denoted in gray, Pt in white, and background/pores in black.

Overall, there is a good correspondence between the test images and predicted segmentations. The location, shape, and size of the Pt NPs correlate with the test images and the texture of the pores inside the -Alumina matrix. One striking observation is that the catalytic Pt NPs are associated with the crystallographic modulations on the broad {110} surfaces of the -Alumina particle. Most of the Pt NPs are found at the apex of the two {111} type facets, as visualized explicitly in the zoomed-in images.

Our model aims to segment accurately the complex surface and bulk microstructure of the -Alumina particle and Pt NPs with a limited amount of annotated ground-truth data. The results presented establish that the U-Net model with a weighted focal loss provides a stable model for the multi-class semantic segmentation of a large data set of 3D reconstructions in a severe class imbalance situation.

Segmentation results establish a basis for the quantification of critical microstructural parameters, including quantification of the external surfaces in terms of their general area and proportion of the individual facets, quantification of the volume fraction of pores, and their surface area, Pt particle attachment, etc. While this topic will be the focus of our future work, an essential qualitative assessment of the -Alumina surfaces and geometry of the Pt NPs, can be obtained by transforming the stack of predicted segmentations into 3D visualizations.

Figure 6. shows 3D volume visualization of -Alumina and Pt NPs from different viewpoints, and surface contour maps in light gray represent -Alumina and red Pt NPs. 3D visualizations show that {110} surfaces of -Alumina are not atomically flat; instead, they form a series of periodically repeating structural facets. These facets are mostly terminated towards the center of -Alumina, and Pt NPs are anchored along with the {111} type facets rather than randomly distributed on the surfaces. Surprisingly, matrix pores are aligned along the direction of the surface facets, as seen in Figure 6.d.

Figure 6: 3D visualization of -Alumina/Pt catalytic particle from different viewpoints.

4 Conclusion

We have demonstrated the effectiveness of a deep learning model in multi-class semantic segmentation of large and unbalanced data. This work was in many respects exploratory as a proof of concept, with a focus mainly on model performance. As such, the current model is naturally limited in extending its applicability. Alumina-based catalysts with various morphologies and particle contrasts will require additional adaptations to the current model. We intend to continue to utilize this U-net model in a transfer learning environment, incorporating the broad morphological and size variation of Alumina-based catalysts and eventually with general catalyst systems to achieve wider applicability. With recent advances in automatic data collection, deep learning-assisted semantic segmentation is genuinely expected to broaden the field of STEM tomography for routine quantitative measurement of catalysts on a statistically relevant scale, which is not possible currently.

Acknowledgement

The experimental work was conducted in the Environmental Molecular Sciences Laboratory (EMSL), a national scientific user facility sponsored by the Department of Energy’s Office of Biological and Environmental Research at Pacific Northwest National Laboratory (PNNL). PNNL is a multi-program national laboratory operated for the DOE by Battelle Memorial Institute under Contract DE-AC05- 76RL01830. LK was supported by the U.S. Department of Energy (DOE), Office of Basic Energy Sciences, Division of Chemical Sciences, Geosciences, and Biosciences.

References

  • [1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. G. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng (2016)

    TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

    .
    ArXiv abs/1603.04467. Cited by: §2.3.
  • [2] J. P. Ahrens, B. Geveci, and C. C. Law (2005)

    ParaView: An End-User Tool for Large-Data Visualization

    .
    In The Visualization Handbook, Cited by: §2.1.
  • [3] I. Arslan, T. J. V. Yates, N. D. Browning, and P. A. Midgley (2005-09) Embedded Nanostructures Revealed in Three Dimensions. Science 309 (5744), pp. 2195–2198 (en). External Links: ISSN 0036-8075, 1095-9203, Link, Document Cited by: §1.
  • [4] D. Ciresan, A. Giusti, L. Gambardella, and J. Schmidhuber (2012)

    Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images

    .
    In Advances in Neural Information Processing Systems, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.), Vol. 25. External Links: Link Cited by: §1.
  • [5] B. L. DeCost, B. Lei, T. Francis, and E. A. Holm (2019-02) High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel. Microscopy and Microanalysis 25 (1), pp. 21–29 (en). External Links: ISSN 1431-9276, 1435-8115, Link, Document Cited by: §1, §1.
  • [6] H. Dong, G. Yang, F. Liu, Y. Mo, and Y. Guo (2017) Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks. ArXiv abs/1705.03820. Cited by: §1.
  • [7] Dong-Gyu Sim, Oh-Kyu Kwon, and Rae-Hong Park (1999-03) Object matching algorithms using robust Hausdorff distance measures. IEEE Transactions on Image Processing 8 (3), pp. 425–429. External Links: ISSN 10577149, Link, Document Cited by: §2.4.
  • [8] T. Epicier, S. Koneti, P. Avenier, A. Cabiac, A. Gay, and L. Roiban (2019) 2D & 3D in situ study of the calcination of Pd nanocatalysts supported on delta-Alumina in an Environmental Transmission Electron Microscope. Catalysis Today 334, pp. 68–78. Note: Scientific Bases for the Preparation of Heterogeneous Catalysts. Highlights of the 12th International Symposium, Louvain-la-Neuve, Belgium, July 8-12, 2018 External Links: ISSN 0920-5861, Link, Document Cited by: §1.
  • [9] X. Glorot, A. Bordes, and Y. Bengio (2011-04) Deep Sparse Rectifier Neural Networks. In

    Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics

    , G. Gordon, D. Dunson, and M. Dudík (Eds.),
    Proceedings of Machine Learning Research, Vol. 15, Fort Lauderdale, FL, USA, pp. 315–323. External Links: Link Cited by: §1.
  • [10] H. Greenspan, B. van Ginneken, and R. M. Summers (2016) Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique. IEEE Transactions on Medical Imaging 35 (5), pp. 1153–1159. External Links: Document Cited by: §1.
  • [11] N. F. Greenwald, G. Miller, E. Moen, A. Kong, A. Kagel, T. Dougherty, C. C. Fullaway, B. J. McIntosh, K. X. Leow, M. S. Schwartz, C. Pavelchek, S. Cui, I. Camplisson, O. Bar-Tal, J. Singh, M. Fong, G. Chaudhry, Z. Abraham, J. Moseley, S. Warshawsky, E. Soon, S. Greenbaum, T. Risom, T. Hollmann, S. C. Bendall, L. Keren, W. Graf, M. Angelo, and D. Van Valen (2021-11) Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature Biotechnology. External Links: ISSN 1546-1696, Link, Document Cited by: §1.
  • [12] D. Gürsoy, F. De Carlo, X. Xiao, and C. Jacobsen (2014-09) TomoPy: a framework for the analysis of synchrotron tomographic data. Journal of synchrotron radiation 21 (Pt 5), pp. 1188–1193 (eng). Note: Edition: 2014/08/01 Publisher: International Union of Crystallography External Links: ISSN 1600-5775, Link, Document Cited by: §2.1.
  • [13] D. Gürsoy, Y. P. Hong, K. He, K. Hujsak, S. Yoo, S. Chen, Y. Li, M. Ge, L. M. Miller, Y. S. Chu, V. De Andrade, K. He, O. Cossairt, A. K. Katsaggelos, and C. Jacobsen (2017-12) Rapid alignment of nanotomography data using joint iterative reconstruction and reprojection. Scientific Reports 7 (1), pp. 11818 (en). External Links: ISSN 2045-2322, Link, Document Cited by: §1.
  • [14] Y. Han, J. Jang, E. Cha, J. Lee, H. Chung, M. Jeong, T. Kim, B. G. Chae, H. G. Kim, S. Jun, S. Hwang, E. Lee, and J. C. Ye (2021-03) Deep learning STEM-EDX tomography of nanocrystals. Nature Machine Intelligence 3 (3), pp. 267–274 (en). External Links: ISSN 2522-5839, Link, Document Cited by: §1.
  • [15] K. He, X. Zhang, S. Ren, and J. Sun (2015)

    Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

    .
    In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034. External Links: Document Cited by: §2.3.
  • [16] A. Hernández-García and P. König (2018) Data augmentation instead of explicit regularization. ArXiv abs/1806.03852. Cited by: §1.
  • [17] J. P. Horwath, D. N. Zakharov, R. Mégret, and E. A. Stach (2020-12) Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. npj Computational Materials 6 (1), pp. 108 (en). External Links: ISSN 2057-3960, Link, Document Cited by: §1.
  • [18] D.P. Huttenlocher, G.A. Klanderman, and W.J. Rucklidge (1993) Comparing images using the Hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence 15 (9), pp. 850–863. External Links: Document Cited by: §2.4.
  • [19] S. Jadon (2020) A survey of loss functions for semantic segmentation. 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pp. 1–7. Cited by: §1.
  • [20] K. Khivantsev, N. R. Jaegers, J. Kwak, J. Szanyi, and L. Kovarik (2021-08) Precise Identification and Characterization of Catalytically Active Sites on the Surface of gamma-Alumina. Angewandte Chemie 133 (32), pp. 17663–17671 (en). External Links: ISSN 0044-8249, 1521-3757, Link, Document Cited by: §1.
  • [21] L. Kovarik, A. Genc, C. Wang, A. Qiu, C. H. F. Peden, J. Szanyi, and J. H. Kwak (2013-01) Tomography and High-Resolution Electron Microscopy Study of Surfaces and Porosity in a Plate-like gamma-Al2O3. The Journal of Physical Chemistry C 117 (1), pp. 179–186. Note: Publisher: American Chemical Societydoi: 10.1021/jp306800h External Links: ISSN 1932-7447, Link, Document Cited by: §1, §2.1.
  • [22] A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012) ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.), Vol. 25. External Links: Link Cited by: §1.
  • [23] B. E. Leach (1983) Applied industrial catalysis Volume 1 Volume 1. Academic Press, New York (English). Note: OCLC: 841170071 External Links: ISBN 978-0-323-15633-2 978-1-299-36535-3, Link Cited by: §1.
  • [24] Y. LeCun, Y. Bengio, and G. Hinton (2015-05) Deep learning. Nature 521 (7553), pp. 436–444 (en). External Links: ISSN 0028-0836, 1476-4687, Link, Document Cited by: §1.
  • [25] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár (2020) Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2), pp. 318–327. External Links: Document Cited by: §1, §1, §2.3.
  • [26] J. Long, E. Shelhamer, and T. Darrell (2015-06) Fully convolutional networks for semantic segmentation. In

    2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    ,
    Boston, MA, USA, pp. 3431–3440. External Links: ISBN 978-1-4673-6964-0, Link, Document Cited by: §1.
  • [27] B. J. Maiseli (2021-09) Hausdorff Distance with Outliers and Noise Resilience Capabilities. SN Computer Science 2 (5), pp. 358 (en). External Links: ISSN 2662-995X, 2661-8907, Link, Document Cited by: §2.4.
  • [28] (2022-01) Make smooth predictions by blending image patches, such as for image segmentation. Vooban. Note: original-date: 2017-08-25T20:26:36Zhttps://github.com/Vooban/Smoothly-Blend-Image-Patches External Links: Link Cited by: §2.3.
  • [29] A. Mavrin (2022-01) Focal Loss. Note: original-date: 2019-09-21T05:54:54Zhttps://github.com/artemmavrin/focal-loss External Links: Link Cited by: §2.3.
  • [30] P. A. Midgley, M. Weyland, T. J. V. Yates, I. Arslan, R. E. Dunin-Borkowski, and J. M. Thomas (2006-09) Nanoscale scanning transmission electron tomography. Journal of Microscopy 223 (Pt 3), pp. 185–190 (eng). External Links: ISSN 0022-2720, Document Cited by: §1.
  • [31] P.A. Midgley and M. Weyland (2003-09) 3D electron microscopy in the physical sciences: the development of Z-contrast and EFTEM tomography. Ultramicroscopy 96 (3-4), pp. 413–431 (en). External Links: ISSN 03043991, Link, Document Cited by: §1, §3.
  • [32] P. A. Midgley, M. Weyland, J. M. Thomas, and B. F. G. Johnson (2001) Z-Contrast tomography: a technique in three-dimensional nanostructural analysis based on Rutherford scattering. Chemical Communications (10), pp. 907–908. External Links: ISSN 13597345, 1364548X, Link, Document Cited by: §1.
  • [33] F. Milletari, N. Navab, and S. Ahmadi (2016-10) V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA, pp. 565–571. External Links: ISBN 978-1-5090-5407-7, Link, Document Cited by: §1.
  • [34] A. A. Novikov, D. Lenis, D. Major, J. Hlaeovka, M. Wimmer, and K. Bühler (2018) Fully Convolutional Architectures for Multiclass Segmentation in Chest Radiographs. IEEE Transactions on Medical Imaging 37, pp. 1865–1876. Cited by: §1.
  • [35] D. M. Pelt, D. Gürsoy, W. J. Palenstijn, J. Sijbers, F. De Carlo, and K. J. Batenburg (2016-05) Integration of TomoPy and the ASTRA toolbox for advanced processing and reconstruction of tomographic synchrotron data. Journal of Synchrotron Radiation 23 (3), pp. 842–849. External Links: ISSN 1600-5775, Link, Document Cited by: §2.1.
  • [36] R. Qin, K. Qiao, L. Wang, L. Zeng, J. Chen, and B. Yan (2018-10) Weighted Focal Loss: An Effective Loss Function to Overcome Unbalance Problem of Chest X-ray14. IOP Conference Series: Materials Science and Engineering 428, pp. 012022. Note: Publisher: IOP Publishing External Links: Link, Document Cited by: §1.
  • [37] Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. 7. External Links: ISSN 2057-3960, Link, Document Cited by: §1.
  • [38] G. Roberts, S. Y. Haile, R. Sainju, D. J. Edwards, B. Hutchinson, and Y. Zhu (2019-09) Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels. Scientific Reports 9 (1), pp. 12744. External Links: ISSN 2045-2322, Link, Document Cited by: §1, §1.
  • [39] L. Roiban, L. Sorbier, C. Pichon, C. Pham-Huu, M. Drillon, and O. Ersen (2012) 3D-TEM investigation of the nanostructure of a delta-Al2O3 catalyst support decorated with Pd nanoparticles. Nanoscale 4 (3), pp. 946–954. Note: Publisher: The Royal Society of Chemistry External Links: Link, Document Cited by: §1.
  • [40] O. Ronneberger, P. Fischer, and T. Brox (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI, Cited by: §1.
  • [41] L. A. Shepp and Y. Vardi (1982-10) Maximum Likelihood Reconstruction for Emission Tomography. IEEE Transactions on Medical Imaging 1 (2), pp. 113–122. External Links: ISSN 0278-0062, 1558-254X, Link, Document Cited by: §1.
  • [42] C. H. Sudre, W. Li, T. K. M. Vercauteren, S. Ourselin, and M. J. Cardoso (2017) Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. Deep learning in medical image analysis and multimodal learning for clinical decision support : Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, held in conjunction with MICCAI 2017 Quebec City, QC,… 2017, pp. 240–248. Cited by: §1.
  • [43] T. Sugino, T. Kawase, S. Onogi, T. Kin, N. Saito, and Y. Nakajima (2021) Loss Weightings for Improving Imbalanced Brain Structure Segmentation Using Fully Convolutional Networks. Healthcare 9 (8). External Links: ISSN 2227-9032, Link, Document Cited by: §1.
  • [44] (2022-01) Surface distance metrics. DeepMind. Note: original-date: 2018-07-19T13:46:05Zhttps://github.com/deepmind/surface-distance External Links: Link Cited by: §2.4.
  • [45] A. A. Taha and A. Hanbury (2015-12) Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging 15 (1), pp. 29 (en). External Links: ISSN 1471-2342, Link, Document Cited by: §2.4.
  • [46] Tomviz for tomographic visualization of nanoscale materials. (en-US). Note: https://tomviz.org/ External Links: Link Cited by: §2.1.
  • [47] C. Wang, G. Ding, Y. Liu, and H. L. Xin (2020-12) 0.7 å Resolution Electron Tomography Enabled by Deep‐Learning‐Aided Information Recovery. Advanced Intelligent Systems 2 (12), pp. 2000152 (en). External Links: ISSN 2640-4567, 2640-4567, Link, Document Cited by: §1.
  • [48] M. Weyland, P. A. Midgley, and J. M. Thomas (2001-08) Electron Tomography of Nanoparticle Catalysts on Porous Supports: A New Technique Based on Rutherford Scattering. The Journal of Physical Chemistry B 105 (33), pp. 7882–7886 (en). External Links: ISSN 1520-6106, 1520-5207, Link, Document Cited by: §1.
  • [49] M. Yeung, E. Sala, C. Schönlieb, and L. Rundo (2021) Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation.. Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society 95, pp. 102026. Cited by: §1.
  • [50] T. Zeng, B. Wu, and S. Ji (2017-03) DeepEM3D: approaching human-level performance on 3D anisotropic EM image segmentation. Bioinformatics 33 (16), pp. 2555–2562. Note: _eprint: https://academic.oup.com/bioinformatics/article-pdf/33/16/2555/25163663/btx188.pdf External Links: ISSN 1367-4803, Link, Document Cited by: §1.