Magnetic Resonance Imaging (MRI) is the preferred imaging modality for non-invasive assessment of cardiac performance, thanks to its lack of ionizing radiation, good soft tissue contrast, and high image quality. Cardiac contractile function parameters such as systolic and diastolic volumes, ejection fraction, and myocardium mass are good indicators of cardiac health, representing reliable diagnostic value. Segmentation of the left ventricle (LV) allows us to compute these cardiac parameters, and also to generate high quality anatomical models for surgical planning, guidance, and regional analysis of the heart. Although manual delineation of the ventricle is considered as the gold-standard, it is time consuming and highly susceptible to inter- and intra-observer variability. Hence, there is a need for fast, robust, and accurate semi- or fully-automatic segmentation algorithms.
Cardiac MR image segmentation techniques can be broadly classified into: 1) no-prior based methods, such as thresholding, edge-detection and linking, and region growing; 2) weak-prior based methods, such as active contours (snakes), level-set, and graph-theoretical models; 3) strong-prior based methods, such as active shape and appearance models, and atlas-based models; and 4) machine learning based methods, such as per pixel classification and convolutional neural network (CNN) based models. A comprehensive review of various cardiac MR image segmentation techniques can be found in.
Recent success of deep learning techniques
in high level computer vision, speech recognition, and natural language processing applications has motivated their use in medical image analysis. Although the early adoption of deep learning in medical image analysis encountered various challenges, such as the limited availability of medical imaging data and associated costly manual annotation, those challenges were circumvented by patch-based training, data augmentation, and transfer learning techniques[3, 4].
Long et al.  were the first to propose a fully convolutional network (FCN) for semantic image segmentation by adapting the contemporary classification networks fine-tuned for the segmentation task, obtaining state-of-the-art results. Several modifications to the FCN architecture and various post-processing schemes have been proposed to improve the semantic segmentation results as summarized in . Notably, the U-Net architecture  with data augmentation has been very successful in medical image segmentation.
While segmentation indirectly enables the computation of various cardiac indices, direct estimation of these quantities from low-dimensional representation of the image have also been proposed in the literature [8, 9, 10]. However, these methods are less interpretable and the correctness of the produced output is often unverifiable, potentially limiting their clinical adoption.
Here we propose a CNN based multi-task learning approach to perform both LV segmentation and cardiac indices estimation simultaneously, such that these related tasks regularize the network, hence improving the network generalization performance. Furthermore, our method increases the interpretablity of the output cardiac indices, as the clinicians can infer its correctness based on the quality of produced segmentation result.
Traditionally, the segmentation of the LV and quantification of the cardiac indices have been performed independently. However, due to a close relation between the two tasks, we identified that learning a CNN model to perform both tasks simultaneously is beneficial in two ways: 1) it forces the network to learn features important for both tasks, hence, reducing the chances of over-fitting to a specific task, improving generalization; 2) the segmentation results can be used as a proxy to identify the reliability of the obtained cardiac indices, and also to perform regional cardiac analysis and surgical planning.
2.1 Data Preprocessing and Augmentation
This study employed 97 de-identified cardiac MRI image datasets from patients suffering from myocardial infarction and impaired LV contraction available as a part of the STACOM Cardiac Atlas Segmentation Challenge project [11, 12] database111http://www.cardiacatlas.org. Cine-MRI images in short-axis and long-axis views are available for each case. The semi-automated myocardium segmentation provided with the dataset served as gold-standard for assessing the proposed segmentation technique. The dataset was divided into 80% training and 20% testing for five-fold cross-validation.
The physical pixel spacing in the short-axis images ranged from 0.7031 to 2.0833 mm. We used SimpleITK  to resample all images to the most common spacing of
mm along both x- and y-axis. The resampled images were center cropped or zero padded to a common resolution ofpixels. We applied two transformations, obtained from the combination of random rotation and translation (by maximum of half the image size along x- and y-axis), to each training image for data augmentation.
2.2 Multi-Task Learning using Uncertainty to Weigh Losses
We estimate the task-dependent uncertainty  for both myocardium segmentation and myocardium area regression via probabilistic modeling. The weights for each task are determined automatically based on the task uncertainties learned during the training .
For a neural network with weights , let be the output for the corresponding input . We model the likelihood for segmentation task as the squashed and scaled version of the model output through a softmax function:
where, is a positive scalar, equivalent to the temperature, for the defined Gibbs/Boltzmann distribution. The magnitude of determines the uniformity of the discrete distribution, and hence relates to the uncertainty of the prediction. The log-likelihood for the segmentation task can be written as:
where is the
’th element of the vector.
Similarly, for the regression task we define our likelihood as a Lapacian distribution with its mean given by the neural network output:
The log-likelihood for regression task can be written as:
where is the neural networks observation noise parameter — capturing the noise in the output.
For a network with two outputs: continuous output modeled with a Laplacian likelihood, and a discrete output modeled with a softmax likelihood, the joint loss is given by:
where is the mean absolute distance (MAD) loss of and is the cross-entropy loss of . To arrive at (5), the two tasks are assumed independent and simplifying assumptions have been made for the softmax likelihood, resulting in a simple optimization objective with improved empirical results . During the training, the joint likelihood loss is optimized with respect to as well as , .
As observed in (5), the uncertainties (, ) learned during the training are weighting the losses for individual tasks, such that, the task with higher uncertainty is weighted less and vice versa. Furthermore, the uncertainties can’t become too large, as they are penalized by the last two terms in (5).
2.3 Network Architecture
In this work, we adapt the U-Net architecture , highly successful in medical image segmentation, to perform an additional task of myocardium area estimation as shown in Fig. 1. The segmentation and regression paths are split at the final up-sampling and concatenation layer. The final feature map in the segmentation path is passed through a sigmoid layer to obtain a per-pixel image segmentation. Similarly, the regression output is obtained by down-sampling the final feature map in the regression path by of its size and passing it through a fully-connected layer. The logarithm of the task uncertainties (, ) added as the network parameters are used to construct the loss function (5), and are learned during the training. Note that we train the network to predict the log uncertainty ) due to its numerical stability and the positivity constraint imposed on the computed uncertainty via exponentiation, .
The network was initialized with the Kaiming uniform 
initializer and trained for 50 epochs usingRMS prop
optimizer with a learning rate of 0.001 (decayed by 0.95 every epoch) in PyTorch222https://github.com/pytorch/pytorch. The best performing network, in terms of the Dice overlap between the obtained and gold-standard segmentation, in the test set, was saved and used for evaluation.
The network training required 9 minutes per epoch on average using a 12GB Nvidia Titan Xp GPU. It takes 0.663 milliseconds on average to process a slice during testing. The log uncertainties learned for the segmentation and regression tasks during training are and , respectively, which correspond to weighting the cross-entropy and mean absolute difference (MAD) loss by a ratio of 1556:1. Note that the scale for cross-entropy loss is , whereas that for MAD loss is .
|Assessment Metric||End-Diastole||End-Systole||All Phases|
|Mean Surface Distance (mm)|
|Hausdorff Distance (mm)|
The 2D segmentation results are stacked to form a 3D volume, and the largest connected component is selected as the final myocardium segmentation. The myocardium segmentation obtained for end-diastole, end-systole, and all cardiac phases from the proposed multi-task network (MTN) and from the baseline U-Net architecture (without the regression task) are both assessed against the gold-standard segmentation provided with the dataset as part of the challenge, using four traditionally employed segmentation metrics — Dice Index, Jaccard Index, Mean surface distance (MSD), and Hausdorff distance (HD) — summarized in Table 1. Note that the myocardium dice coefficient is higher for end-systole phase where the myocardium is thickest.
The Kolmogorov-Smirnov test shows that the difference in distributions for Dice, Jaccard and MSD metrics between the proposed multi-task network and baseline U-Net architecture are statistically significant with p-values: , , and , respectively. However, since the segmentation is evaluated on a large sample of 2191 volumes across five-fold cross validation, the p-values quickly go to zero even for slight difference in distributions being compared, representing no practical significance . Hence, we computed the 99% confidence interval for the mean value of each segmentation metric based on 1000 bootstrap re-sampling with replacement, as shown in Fig. 2. As evident from Fig. 2, Dice, Jaccard and HD metrics are statistically similar, whereas the reduction in MSD for the proposed multi-task network compared to the baseline U-Net architecture is statistically significant.
In addition to obtaining the myocardium area from the regression path of the proposed network, it can also be computed indirectly from the obtained myocardium segmentation. Hence, we compute and evaluate the myocardium area estimated from three different sources: (a) regression path of the MTN, (b) segmentation obtained from the MTN, and (c) segmentation obtained from the baseline U-Net model. Fig. 3 shows the myocardium area obtained from these three methods for all phases of the cardiac cycle plotted against the ground-truth myocardium area estimated from the gold-standard myocardium segmentation provided as part of the challenge data. We can observe a linear relationship between the computed and gold-standard myocardium areas, and the corresponding correlation coefficients for the methods (a), (b), and (c) are 0.9466, 0.9565, 0.9518, respectively.
|Cardiac Regions||End-Diastole||End-Systole||All Phases|
(a) Box-plot (outliers removed for clarity) and (b) Mean and 99% confidence interval, for the mean absolute difference (MAD) between the myocardium area obtained from the provided gold-standard segmentation and the results obtained from: (1) the regression path of the proposed multi-task network, (2) segmentation obtained from the proposed multi-task network, and (3) segmentation obtained from the baseline U-Net model. Confidence intervals are obtained based on 1000 bootstrap re-sampling with replacement.
Further, we computed the MAD between the ground-truth myocardium area and the area estimated by each of the three methods for end-diastole, end-systole, and all cardiac phases (for 26664 slices across five-fold cross validation). For the regional analysis, slices in the ground-truth segmentation after excluding two apical and two basal slices are considered as mid-slices. Table 2
summarizes the mean and standard deviation for the computed MADs. Box-plots (outliers removed for clarity) comparing the three methods for different regions of the heart throughout the cardiac cycle are shown inFig. (a)a. The MAD in myocardium area estimation of mm obtained from the regression output of the proposed method is similar to the results presented in : mm, while acknowledging the limitation that the study in  was conducted on a different dataset than our study. Moreover, while the regression output of the proposed network yields good estimate of the myocardial area, the box-plot in Fig. (a)a suggests that even further improved myocardial area estimates can be obtained from a segmentation based method, provided that the quality of the segmentation is good.
Lastly, we computed the 99% confidence interval for the mean value of myocardium area MAD based on 1000 bootstrap re-sampling with replacement, as shown in Fig. (b)b. This confirms that the myocardium area estimated from the segmentation output of the proposed multi-task network is significantly better than that obtained from the regression output, however, there is no statistical significance between other methods. Furthermore, we can observe the variability in MAD is highest for the basal slices, followed by apical and mid slices.
4 Discussion, Conclusion, and Future Work
We presented a multi-task learning approach to simultaneously segment and quantify myocardial area. We adapt the U-Net architecture, highly successful in medical image segmentation, to perform an additional regression task. The best location to incorporate the regression path into the network is a hyper-parameter, tuned empirically. We found that adding the regression path in the bottleneck or intermediate decoder layers is detrimental for the segmentation performance of the network, likely due to high influence of the skip connections in the U-Net architecture.
Myocardium area estimates obtained from the regression path of the proposed network are similar to the direct estimation-based results found in the literature. However, our experiments suggest that segmentation-based myocardium area estimation is superior to that obtained from a direct estimation-based method. Lastly, the myocardium segmentation obtained from our method is at least as good as the segmentation obtained from the baseline U-Net model.
To test the generalization performance of the proposed multi-task network, we plan to evaluate the network performance using a lower number of training images. Similarly, we plan to extend this work to segment left ventricle myocardium, blood-pool, and right ventricle, and regress their corresponding areas using the Automated Cardiac Diagnosis Challenge (ACDC)333https://www.creatis.insa-lyon.fr/Challenge/acdc/ 2017 dataset.
Research reported in this publication was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award No. R35GM128877 and by the Office of Advanced Cyberinfrastructure of the National Science Foundation under Award No. 1808530. Ziv Yaniv’s work was supported by the Intramural Research Program of the U.S. National Institutes of Health, National Library of Medicine.
-  Petitjean, C., Dacher, J.N.: A review of segmentation methods in short axis cardiac MR images. Medical Image Analysis 15(2) (2011) 169 – 184
-  LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553) (2015) 436–444
-  Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Annual review of biomedical engineering 19 (2017) 221–248
-  Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A., van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical Image Analysis 42 (2017) 60 – 88
-  Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: The IEEE Conference on CVPR. (2015)
-  Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Rodríguez, J.G.: A review on deep learning techniques applied to semantic segmentation. CoRR abs/1704.06857 (2017)
-  Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. CoRR abs/1505.04597 (2015)
-  Zhen, X., Islam, A., Bhaduri, M., Chan, I., Li, S.: Direct and simultaneous four-chamber volume estimation by multi-output regression. In Navab, N., Hornegger, J., Wells, W.M., Frangi, A., eds.: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham, Springer International Publishing (2015) 669–676
-  Xue, W., Islam, A., Bhaduri, M., Li, S.: Direct multitype cardiac indices estimation via joint representation and regression learning. IEEE Transactions on Medical Imaging 36(10) (2017) 2057–2067
-  Xue, W., Brahm, G., Pandey, S., Leung, S., Li, S.: Full left ventricle quantification via deep multitask relationships learning. Medical Image Analysis 43 (2018) 54 – 65
-  Fonseca, C.G., Backhaus, M., Bluemke, D.A., Britten, R.D., Chung, J.D., Cowan, B.R., Dinov, I.D., Finn, J.P., Hunter, P.J., Kadish, A.H., Lee, D.C., Lima, J.A.C., Medrano−Gracia, P., Shivkumar, K., Suinesiaputra, A., Tao, W., Young, A.A.: The cardiac atlas project - an imaging database for computational modeling and statistical atlases of the heart. Bioinformatics 27(16) (2011) 2288–2295
-  Suinesiaputra, A., Cowan, B.R., Al-Agamy, A.O., Elattar, M.A., Ayache, N., Fahmy, A.S., Khalifa, A.M., Medrano-Gracia, P., Jolly, M.P., Kadish, A.H., Lee, D.C., Margeta, J., Warfield, S.K., Young, A.A.: A collaborative resource to build consensus for automated left ventricular segmentation of cardiac MR images. Medical Image Analysis 18(1) (2014) 50 – 62
-  Yaniv, Z., Lowekamp, B.C., Johnson, H.J., Beare, R.: SimpleITK image-analysis notebooks: a collaborative environment for education and reproducible research. Journal of Digital Imaging (2017)
-  Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., eds.: Advances in Neural Information Processing Systems 30. Curran Associates, Inc. (2017) 5574–5584
-  Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. CoRR abs/1705.07115 (2017)
He, K., Zhang, X., Ren, S., Sun, J.:
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.In: Proceedings of the IEEE international conference on computer vision. (2015) 1026–1034
-  Lin, M., Lucas Jr, H.C., Shmueli, G.: Research commentary - too big to fail: large samples and the p-value problem. Information Systems Research 24(4) (2013) 906–917