Brain Magnetic Resonance Images (MRIs) allow for three-dimensional (3D) imaging of the brain and are widely used in research and clinical practice for the diagnosis and treatment of neurological diseases. While promising technology advancements of the imaging quality enable an ever-increasing amount of conditions that become detectable vernooij2007incidental, reading and interpreting MRI remains a challenging task. First, brain lesion detection and delineation requires expert knowledge and is a tedious time-consuming process, affected by human errors bruno2015understanding. Second, MRI is increasingly used and hence an ever-increasing amount of images need to be studied, while only a limited number of experts are available chen2018unsupervised. This leads to the urgent need for automatic detection and segmentation of lesions to assist radiologists during clinical practice.
Recently, supervised deep learning methods have shown promising results for this task, while the success of these methods depends heavily on large data sets with high-quality annotations lundervold2019overview. Note that supervised methods only generalize well to cases that are sufficiently represented in the training data. However, diverse and large annotated data sets are costly to obtain, and often only a few limited cases are available for rare diseases baur2020autoencoders.
In contrast to that, human experts can be trained with few healthy cases to generalize, and afterwards they are able to detect even arbitrary anomalies without being trained to an explicit appearance chen2018unsupervised
. Deep learning for unsupervised anomaly detection (UAD) follows this concept of identifying unexpected, abnormal data. These methods do not require pixel-level annotations and are only trained with MRI-scans of healthy brains. Here, the task is considered as an anomaly detection problem, where the networks are trained to represent the distribution of healthy anatomy of the human brain and anomalies can be detected as outliers from the learned distribution. Typically, deep learning for UAD follows an encoder-decoder structure trained only on healthy images. Afterwards, detection and delineation of pathologies of a test image can be obtained, e.g., by pixel-wise discrepancies between the model’s input and reconstruction.
So far, a wide range of deep learning methods have been proposed for UAD in brain MRI, ranging from simple auto-encoders baur2018deep
to generative adversarial networks (GANs)schlegl2019f focusing on 2D spatial information. These 2D methods have shown promising results, however, the global spatial context provided by MRI volumes remains unused and the inherently 3D structure of brains cannot be learned by the networks. This brings up the question, whether increased spatial context by using entire MRI volumes allows for improved performance, leading to the problem of 3D deep learning for UAD in brain MRI. So far, 3D deep learning for UAD has hardly been considered, only pioneering work in volumetric head CT data has been proposed recently without direct comparison to 2D sato2018primitive
. 3D deep learning is challenging in nature as it results in an increased representational power that may come with an increased risk of overfitting, leading to poor generalization. For preventing the risk of overfitting several different regularization strategies have been proposed for deep learning in the context of computer vision. These methods range from simple image transformation such as rotation and flipping to adding noise during the training process, e.g., by stochastically dropping out neuron activationssrivastava2014dropout or dropping out entire input regions devries2017improved during training. Especially the latter has been combined with 2D auto-encoder networks, called context-encoders pathak2016context, where the networks are enforced to generate the contents of an arbitrary image region conditioned on its surroundings, leading to a better understanding of the global content of the image. This idea has also shown promising results in the context of UAD in brain MRI using 2D methods zimmerer2018context and might be a promising approach for enforcing the understanding of the global context when entire MRI volumes are used in combination with 3D deep learning.
In this paper, we propose to learn from entire 3D MRI volumes instead of single 2D MRI slices using 3D instead of 2D unsupervised deep learning, shown in Figure 1. Also, we extend the concept of spatial input erasing for regularization. To this end, we provide an extensive comparison of variational autoencoders (VAE) with 3D and 2D convolutions and propose several different 3D spatial erasing strategies during training. For our experiments, we use a training data set with brain MRI scans of 2008 healthy patients and evaluate our methods on two publicly available brain segmentation data sets. We focus on T1-weighted MRI data, which is widely used in clinics ito2019comparison; akkus2017deep, providing a good starting point for anomaly detection. Moreover, we provide an analysis of the impact and the importance of the training data set size, especially in combination with our 3D approach.
2 Materials and Methods
2.1 Data Set
For training, we consider a data set with anonymized T1-weighted MRI volumes of 2008 healthy subjects from 22 scanners from different vendors. The resolutions in axial direction vary from to with a majority of 1310 samples with . The slice thickness lies between to with a majority of 906 samples with . 1506 samples, are acquired with a field strength of 1.5 T, 446 samples are acquired with 3 T and 56 with 1 T.
Data on all scanners was acquired during clinical routine with a standard 3D gradient echo sequence. All scans were sent to jung diagnostics GmbH for image analysis.
For evaluation, we use two publicly available data sets. First, we consider the publicly available Multimodal Brain Tumor Segmentation Challenge 2019 (BraTS 2019) data set menze2014multimodal; bakas2017advancing; bakas2018identifying with T1-weighted image volumes of 335 subjects with the corresponding ground truth segmentation of the tumor. The slice thickness of the BraTS 2019 data set varies from up to . Second, we use the Anatomical Tracings of Lesions After Stroke (ATLAS) data set liew2018large, which provides T1-weighted image volumes of 304 subjects with corresponding ground truth segmentations of stroke regions. The slice thickness of the ATLAS data set varies from up to .
For all image volumes, we apply the following preprocessing. First, we resample all scans to the same isotropic resolution of
using cubic interpolation. Then, we follow the preprocessing of previous studies with 2D deep learning methods for UAD, which include skull stripping, denoising, and standardizationbaur2020autoencoders
. Next, we crop excessive background by using brain masks of the MRI scans and zero-pad all MRI scans to the largest volume resolution in our data set of. Last, we downsample all volumes to a size of for numerical efficiency, as we encounter the computational complexity of 3D deep learning. Regarding our data split for training, we consider 1807 healthy images for training and 201 images for validation of our reconstruction performance. We split our data randomly and stratified by scanners. Considering the images of the BraTS 2019 data set, we randomly sample 133 images for validation and 202 for testing. Using the ATLAS data set, we randomly sample 121 and 183 images for validation and testing, respectively.
2.2 Deep Learning Methods
We address the problem of anomaly segmentation with 2D and 3D unsupervised deep learning methods using 2D MRI slices or 3D MRI volumes, respectively. Given a set of healthy MRI scans, we utilize an encoder-decoder architecture and train our methods to encode to and reconstruct from a lower-dimensional latent space . After the methods are trained, anomalies in a test image can be detected by large reconstruction errors between the input and output image, as the networks are trained to reconstruct only images of healthy brain anatomies, e.g., fail to reconstruct abnormal image areas.
Recently, a comparative study on UAD using 2D deep learning methods baur2020autoencoders has demonstrated that VAE kingma2013auto; baur2018deep
allow for promising results, while also being easy to optimize and involving fewer hyperparameters compared to other UAD methods such as GANs. Comparing the VAE with the standard AE, the VAE enforces a structure on the manifold. It has been demonstrated that this leads to performance improvements compared to the standard AEbaur2020autoencoders. Hence, we consider the concept of VAEs for our study.
Our general backbone network is shown in Figure 2 and for the adaption to 2D MRI slices or 3D MRI volumes, we employ 2D or 3D operations for the network, e.g., we use 2D or 3D convolutions. In this way, the architecture details remain the same for 2D and 3D, e.g., the number of layers and feature maps remain same, and only the dimension of the networks operation are changed. Based on our validation set performance, we choose a latent space size of and for our 2D and 3D VAE, respectively.
We study and extend the concept of cutout devries2017improved and context-autoencoders pathak2016context, which were proposed for 2D images. The main motivation behind our approach is to further enhance the usage of global image context, especially in combination with 3D methods. Therefore, we propose and evaluate the following different erasing methods for 2D and 3D, which are shown in Figure 3. Note, we only erase the regions in the input image and not in the ground-truth image that is used for optimization, hence our networks are enforced to solve an in-painting task for abnormal regions.
First, we simply mask-out a single patch in the input, similar to previous concepts for 2D problems devries2017improved; pathak2016context; zimmerer2018context. Also, we extend this approach to 3D and mask-out a single 3D cube. For the patch and cube erasing method, we randomly select a pixel coordinate within the image as a center point and randomly erase regions with a size from up to of the input size. Note, we refer to this method as patch for 2D and cube for 3D.
Second, we extend this approach and split a single patch or cube into multiple ones. To this end, we mask-out up to ten randomly located and sized patches or cubes within an input image, while the overall erasing size remains in the limit of up to of the input size. We call this method multiple-patch or multiple-cube for 2D and 3D, respectively.
Third, we erase entire brain sides based on the idea of stimulating the networks to exploit the symmetry of a brain. Hence, we randomly erase the right or left side of the brain in the input slice. Similar for 3D, here we randomly erase the right or left side of the brain in 1 up to 32 multiple sequential input slices. We refer to this method as half-slice for 2D and half-volume for 3D.
We systematically evaluate all erasing methods with different strategies for masking-out the regions. First, we simply erase regions in the input, e.g., all intensity values of a region are set to zero similar to previous works devries2017improved; pathak2016context; zimmerer2018context
. Second, to further increase the variance of our erasing methods we fill the erased region with noise sampled from the image pixel distribution.
For all our methods, we set the probability of the spatial erasing to, such that the network still receives unmodified images.
2.3 Training and Evaluation
We follow the idea of VAEs, hence we optimize our networks with respect to the reconstruction loss between the original input image and the network output reconstruction combined with the constraint that the latent variables follow a multivariate normal distribution. Hence, our loss function is based on the
-distance between our input and output combined with the distribution-matching Kullback–Leibler divergence for regularization. We train our networks with a batch size of 32 using Adam for optimization with a learning rate of
. We individually tune the number of training epochs of the networks using the reconstruction performance on our validation set with images of healthy subjects.
For all evaluations, we employ the following post-processing steps. First, we multiply each residual image by a slightly eroded brain mask to account for errors occurring at sharp brain-mask boundaries. Next, we remove small outliers with a median filter. For anomaly segmentation of a test image, we consider the voxel-wise residuals obtained from the -distance between the original input image and the network’s reconstruction.
For comparison of our methods, we consider voxel-wise anomaly segmentation performance. To this end, we consider the Dice coefficient (DICE) which is defined by
with two sets and
. Noteworthy, evaluating the DICE requires binarization of the difference image between the original input image and the network’s reconstruction. For this purpose, we utilize our validation set and perform a greedy search to determine the binarization threshold for the segmentation, similar tobaur2020autoencoders
. Since the scans are normalized, intensity intervals range from 0 to 1. Using the ground truth segmentation, we compute the DICE on the validation set for thresholds at the upper and lower quartile of the center of the intensity interval. Based on the DICE we cut the interval to either the lower or upper half and continue the search with the updated interval. The procedure is repeated for 10 iterations and we use the binarization threshold that leads to the best DICE score. Afterwards, we use the determined binarization threshold for the test sets. We report the DICE on an entire data set (DICE
), and also report mean and standard deviation for the subject-wise values (DICE). Moreover, to evaluate the models performance for different operating points, e.g., binarization threshold for segmentation, we also consider the area under the Precision-Recall-Curve (AUPRC). Here, for each data set, we generate Precision-Recall-Curves (PRC) for each model and then we compute the area under it (AUPRC).
Moreover, we consider our best performing methods and our baseline methods with respect to slice-wise anomaly detection. This allows for localization of anomalies on a slice-level in a volume, i.e., which slice contains a lesion. For this purpose, we divided each volume in our test set into normal and abnormal slices. Considering the lesion annotations, we strictly consider all slices with annotations as abnormal and normal otherwise. For discrimination between normal and abnormal slices, we use the -distance between the original input and the network’s reconstruction calculated for each slice. For evaluation of our slice-wise anomaly detection performance independent of the operating point we report the AUPRC.
First, we compare 2D and 3D UAD deep learning methods combined with our erasing regularization methods in Table 1. For both VAEs, our different erasing methods lead to performance improvements. Overall, our 3D VAE outperforms the 2D VAE for all our experiments. Using noise for masking-out the regions works slightly better than masking-out with zeros. For our 3D VAE using a single cube for erasing, followed by masking-out an entire brain side in a subvolume works best. Considering our 2D-VAE, masking-out an entire brain side shows the best results, closely followed by masking-out multiple patches. Comparing the DICE of our best performing 3D approach (3D-cube-n) with the 2D baseline approach (2D-None) demonstrates a relative performance improvement of and on the BraTS 2019 and ATLAS data set, respectively.
Second, we evaluate the performance of our baselines and best performing methods with respect to lesion size in Figure 4. Here, our results demonstrate that the smallest and largest lesions are challenging. Consistently, using erasing improves the DICE over all lesion sizes, while being particularly effective for large lesions. Also, comparing 2D and 3D methods shows that 3D consistently outperforms 2D, especially for small lesions.
Third, we evaluate the effect of the data set size in Figure 5. Reducing the data set has a pronounced impact on the performance for 3D as well as 2D, especially when less than 60% of the training data is used. Also, the spatial erasing works better when the network is trained with more data. While reducing the data set size has a larger impact on 3D, even with only 20% of the training data the 3D VAE works better than the 2D VAE with erasing and 100% of the training data. Moreover, our erasing turns out to be effective for the 2D VAE, considering that a 2D VAE without erasing trained with 100% of data is outperformed by a 2D VAE with erasing trained with only 20% of the data.
Fourth, Figure 7 demonstrates example images for our best performing method 3D-Cube-n. Notably, the ground truth segmentation are highlighted in all difference images, while also showing errors at further regions.
Moreover, we use our best performing 2D and 3D methods trained on T1-weighted MRI data and evaluate on T1ce-weighed MRI data from the BraTS 2019 data set to study the effect of using additional image information, see Table 2. Here, we observe immediate performance improvements compared to T1-weighting for both 2D and 3D with a relative improvement of 13.61% and 21.82% for 2D and 3D considering the DICE.
Last, we evaluate our baseline and best performing methods with respect to slice-wise anomaly detection, see Figure 6. Here, our best performing method achieves an AUPRC of 71.2%. Also for this task using 3D information and erasing turns out to be beneficial, improving the AUPRC by approximately 4% compared to the 2D VAE.
|Input & Erasing||DICE||DICE ()||AUPRC|
|Input & Erasing||DICE||DICE ()||AUPRC|
|Input & Erasing||Sequence||DICE||DICE ()||AUPRC|
We consider the problem of unsupervised anomaly segmentation and propose to learn from entire 3D MRI volumes instead of single 2D MRI. For this purpose, we extend 2D VAEs to 3D and also propose several different input erasing methods for regularization.
Comparing our 2D VAE (2D-None) with the corresponding 3D version (3D-None) without any input erasing demonstrates that 3D outperforms the 2D version on two public data sets, especially for the stroke data set with a DICE of for 3D compared to a DICE of for 2D, see Table 1. This highlights that 3D information can be effectively leveraged by a 3D VAE and agrees with our expectation that increased spatial context by using entire MRI volumes allows for improved anomaly segmentation performance.
We also evaluate 2D and 3D input erasing for regularization and train the networks to restore missing image parts conditioned on its surroundings. Our results in Table 1 demonstrate that input erasing allows for further performance improvements both for our 2D and 3D VAE. Regarding the method for masking-out a region, previous works in 2D mostly simply mask our input regions with zeros zimmerer2018context; pathak2016context; devries2017improved. However, our results demonstrate that using noise for masking-out a region in the input works slightly better, indicating that the increased variance during training is advantageous for regularization.
We also consider different strategies such as erasing multiple patches or an entire brain side. While all erasing strategies are beneficial, there is no clear winner between the different strategies considering our results on both data sets. Furthermore, one could argue that our input erasing leads to brain anatomy that deviates from normal, which is in slight contrast to the idea of only providing healthy brain anatomy as input. However, our ground-truth image that is used for optimization remains unmodified, hence our networks are enforced to solve an in-painting task for abnormal regions. Our results demonstrate that this leads to an improved segmentation performance.
To gain further insights, we study the performance with respect to the lesion size in Figure 4. While providing consistent performance improvements, erasing turns out to be especially valuable for larger lesions. This might be attributed to the fact that with erasing, networks are enforced to solve an additional in-painting task, making them suited to handle inputs with large anomalies. Also, our results in Figure 4 further emphasize the value of 3D information, especially for smaller lesions considering the ATLAS data set.
Next, we study the effect of the training data set size. As expected, the data set size has a notable impact on the performance, see Figure 5. It stands out that our 3D methods trained with only 20% of the training data even outperform the 2D methods trained with 100% of the data. This indicates that increasing the spatial context during training is even more important than increasing the data set size. This is an interesting observation, as one could assume that due to the increased number of parameters, 3D-Models require more data compared to their 2D-counterparts. We believe that this counter-intuitive behaviour could explained by the increased complexity of the task and the bigger input image for the 3D approach. The learning task of the 3D model can be considered more complex since an entire volume must be processed and reconstructed at once, while 2D is only trained to process a single slice. Also, for 3D the input image is bigger (volume) compared to 2D (single slice). Note, if the input image is bigger, then a network might need more expressive power to capture the patterns in the input image, as shown in tan2019efficientnet.
Considering our erasing approach and the data set size suggests that solving the additional in-painting task needs sufficient training data to provide effective regularization. However, with only 60% of the training data our models with our regularization approach lead to higher performance than a model without regularization trained with the full dataset. We argue this demonstrates the effectiveness of our regularization approach, as less data is required to achieve similar or better performance compared to a model without regularization. Still, increasing the data set size is valuable as the performance for our model with erasing continues to improve with a larger training data set.
Comparing our novel 3D methods with input erasing with the previous 2D approach demonstrates a relative performance improvement of and on the BraTS 2019 and ATLAS data set, respectively. A comparable work evaluating UAD performance on the same ATLAS data set achieves a mean subject-wise DICE score of with their best performing method chen2020unsupervised. Notably, this 2D method is restoration-based and involves significantly increased computational complexity. Our 3D approach with input erasing leads to a mean subject-wise DICE score of , improving the UAD state-of-the-art on this data set. This demonstrates the effectiveness of our approach. Comparing our results on the BraTS 2019 data set with other works that utilize additional image information, e.g. T2-weighted data chen2020unsupervised; zimmerer2018context, highlights the advantage of additional image information. Similar, we observe immediate performance improvement for our methods when evaluated on T1ce-weighted data, despite the domain adaption from T1, see Table 2. Also, other studies that use multiple MRI sequences baur2020autoencoders; baur2018deep achieve higher performance metrics, however, a direct comparison is difficult due to different data sets and settings. Notably, multiple MRI sequences are beneficial but not always available ito2019comparison; akkus2017deep, imposing an additional challenge on UAD.
Putting UAD into perspective with supervised methods demonstrates that segmentation performance is in a moderate range. Considering the BRATS 2019 data set, supervised methods achieve a mean subject-wise DICE score of around 90% jiang2019two utilizing all available MRI sequences (T1, T1ce, T2, FLAIR). Considering the ATLAS data set, supervised methods achieve mean subject-wise DICE scores in the range of 32.92% up to 53.49% ito2019comparison. While UAD is notably more challenging than supervised segmentation, the overall UAD performance on these supervised data sets might also be limited, as the annotation focuses on pre-specified lesions and not all anomalies in the images might be labeled. This is also demonstrated in Figure 7, where, e.g., the segmentation focuses only on the tumor and not on all brain regions that deviate from normal. Also, the domain shifts between different data sets might be challenging, which is also pointed out in previous works baur2020autoencoders; zimmerer2018context.
Considering these challenges we also evaluate our methods with respect to slice-wise anomaly detection, see Figure 6. Here, we observe significantly increased performance compared to segmentation with an AUPRC of 71.2% for our best performing method. The slice-wise detection performance motivates that UAD can be helpful in red-flagging suspicious MRI data in clinical routine, especially with T1-weighted MRI data. Also, we believe that unsupervised segmentation gives additional cues to the reader as to where an anomaly may be located and thus it is helpful to quickly localize a potential anomaly or lesion. For this our work consist a valuable contribution by demonstrating the benefits and emphasizing the use of 3D-models with spatial erasing for voxel-wise and slice-wise UAD.
For future work, our findings could be extended to more complex deep learning methods for UAD, such as GANs schlegl2019f. In particular, combining our 3D approach with restoration-based methods chen2020unsupervised might improve the overall performance. However, this approach also leads to significantly increased runtime and computational efforts, e.g., a restoration accumulates quickly to multiple minutes for a single MRI baur2020autoencoders, which is particularly challenging for clinical routine.
We study the task of unsupervised anomaly segmentation in brain MRI and propose to use entire 3D MRI volumes instead of single 2D MRI slices by extending 2D VAEs to 3D. Also, we study and extend the concept of input erasing and propose several different 3D input erasing strategies for regularization. Overall, our results demonstrate that using increased spatial context by using entire MRI volumes combined with 3D deep learning clearly outperforms 2D methods. Also, we observe that combining deep learning with spatial input erasing allows for further performance improvements and reduces the requirement for large training data sets.
Compliance with ethical standards
Funding: This work was partially funded by Grant Number ZF4026303TS9
Conflict of interest: The authors declare that they have no conflict of interest.
Ethical approval: This work was conducted retrospectively on data from clinical routine which was completely anonymized. Ethical approval was therefore not required. Also this work relies on the BraTS 2019 and ATLAS data set. For use of these data sets, no ethics statements are necessary.
Informed consent: Not applicable