Treatment of breast cancer consists of surgery, often followed by radiotherapy and systemic therapy. Systemic therapy — and particularly chemotherapy — can be administered prior to surgery. This so-called neoadjuvant chemotherapy (NAC) allows shrinking of the tumor before surgery, reducing surgical extent, complications, and improving cosmetic outcome [1, 2, 3]. ††This work has been accepted for SPIE Medical Imaging 2019, Computer-Aided Diagnosis conference, Paper 10950-12.
The response of the tumor to NAC can be monitored by imaging the tumor . The change in visible tumor volume on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) can be used as a measure of response . Consequently, the NAC treatment plan can be adjusted when tumor response is deemed insufficient.
In our current response monitoring workflow, seed points are manually placed in areas of enhancement that are likely to contain tumor. A constrained volume growing method uses these manually placed seed points as input and generates a tumor segmentation . This method was rigorously validated using complete pathological embedding . Even though the method is very accurate , manual placement of the required seed points is a time consuming step, prone to intraobserver and interobserver variability. Therefore, automating placement of these seed points is of interest.
Deep learning algorithms have shown high robustness in a variety of medical image analysis tasks . In breast MRI, these methods have mainly focused on diagnosis or on segmentation of healthy tissues [9, 10, 11]. In this study, we propose to exploit deep learning for fast and automatic seed point detection, replacing manual seed point placement in our existing and well-validated workflow [6, 7].
2 Material and Methods
We developed the automatic seed point generator using an ensemble of three independently trained fully convolutional dilated neural networks. Two patient cohorts were used in the study: Cohort 1 consisted of early breast cancer patients in which the seed point detector was developed and evaluated, Cohort 2 consisted of patients receiving NAC in which seed points were automatically placed for constrained volume growing. Tumor response to NAC was evaluated by comparing the change in tumor volume to our reference standard at the MRI before NAC and the MRI during NAC. This reference standard consisted of semi-automatically obtained tumor segmentations. The study design is illustrated in Figure 1, the following paragraphs describe these steps in detail.
Cohort 1: The seed point generator was developed in a subset of 100 randomly selected patients with invasive ductal carcinoma from the prospective Multimodality Analysis and Radiological Guidance IN breast conServing therapy study (MARGINS, 2000-2008) . Patients who were eligible for breast conserving therapy on the basis of conventional imaging and clinical assessment were consecutively recruited for an additional preoperative breast MRI.
Cohort 2: The constrained volume growing was applied to generated seed points in 10 patients with invasive breast cancer who received NAC who particitpated in a prospective studie (2008-2013) . Eligibility criteria included primary invasive breast cancer of 3 cm or larger and/or one or more tumor-positive axillary lymph node.
The baseline characteristics of the patient cohorts are displayed in Table 1.
|Cohort 1 (N = 100)||Cohort 2 (N = 10)||P-value|
|57 (32 – 84)||42 (29 – 64)||0.009|
|Largest tumor diameter on MRI (mm)||21 (9 – 38)||37 (25 – 59)||0.001|
|ER-positive/HER2-negative||73 (73)||4 (40)|
|HER2-positive||15 (15)||0 (0)|
|Triple negative||12 (12)||6 (60)|
|Grade I||30 (30)||0 (0)|
|Grade II||39 (39)||1 (10)|
|Grade III||31 (31)||9 (90)|
|Tumor-positive axillary lymph nodes||0.116|
|0||61 (61)||3 (30)|
|1–3||31 (31)||6 (60)|
|4 or more||8 (8)||1 (10)|
Values are number of patients with percentages in parentheses, unless specified otherwise.
Values are medians, with ranges in parentheses.
Significance levels for differences between patient cohorts are calculated using the Kruskal-Wallis test for
continuous variables and using Fisher’s exact test for categorical variables.
ER = estrogen receptor, HER2 = human epidermal growth factor receptor 2.
2.2 Magnetic resonance imaging
Cohort 1: The patients received one MRI — before surgery. MR images were acquired using a 1.5 T imaging unit (Magnetom; Siemens, Erlangen, Germany) with a dedicated double breast array coil. Five T1-weighted images were acquired: one precontrast image and four postcontrast images after injection of gadolinium-based contrast material (Prohance; Bracco-Byk Gulden, Konstanz, Germany). The imaging parameters were: repetition time 8.1 ms, echo time 4.0 ms, flip angle , isotropic voxel size 1.35 1.35 1.35 mm3, and field of view 310 mm.
Cohort 2: The patients received an MRI before NAC and one after six weeks of NAC. MR images were acquired using a 3 T imaging unit (Achieva; Philips Medical Systems, Best, the Netherlands) with a 7-elements sense breast coil. Six T1-weighted images were acquired: one precontrast image and five postcontrast images after injection of gadolinium-based contrast material (Prohance; Bracco-Byk Gulden, Konstanz, Germany). The imaging parameters were: repetition time 4.4 ms, echo time 2.3 ms, flip angle , isotropic voxel size 1.1 1.1 1.1 mm3, and field of view 360 mm.
Seed point detection was achieved using a dilated convolutional neural network. The convolutional neural network analyzed two-dimensional images with three channels (Figure 2). The first channel consisted of the precontrast image to feed anatomical knowledge to the system. The precontrast images were corrected for field inhomogeneities, and normalized between zero and one on the 5th and 95th intensity percentiles under the breast mask. The second channel consisted of the washin: the relative change in image intensities between the precontrast and the first postcontrast scan . The third channel consisted of the washout: the relative change in image intensities between the first postcontrast and the last postcontrast scan . The washin and washout are by definition normalized measures . Tumors tend to show high washin and high negative washout .
The architecture of the network is shown in Figure 3. The network had a receptive field of 97 97 voxels . Due to increasing dilation factors in subsequent convolutional layers, our network with nine two-dimensional convolutional layers resulted in only
trainable weights. Rectified linear units were used for activation, except for the final layer, which had a binary softmax output. Because the network was fully convolutional, it could be applied to images of arbitrary size. Of the 100 patients, 60 were used for training of the network, 20 for validation, and 20 for independent testing.
An ensemble of three different random initializations of the above-described network was trained in minibatches. Minibatches consisted of 25 patches of 99 99, equally balanced to contain tumor or not. The best model was chosen based on the validation loss between the th and th
iteration. Each network was trained with a different order of the training images. During application of the trained networks, posterior probabilities were thresholded and consensus voting was used on the resulting segmentation maps. Consensus results were used to generate seed points by applying an inward Euclidean distance transform and subsequent local maxima extraction.
Finally, tumor segmentation was performed using our constrained volume growing method that has been validated using complete pathological embedding . In short, starting from a seed point in or near the tumor, the method enhanced the tumor, constructed an ellipsoidal volume of interest around the tumor, and applied a voxel-value based stopping criterion automatically derived from the enhancing tumor .
The detection performance of the automatic seed point generator was evaluated in Cohort 1 using free-response receiver operating characteristics. The seed point generator was subsequently applied to Cohort 2 to assess its application in a response monitoring workflow. The change in tumor volume between the MRI before NAC and the MRI during NAC was calculated, and compared with our reference standard that consisted of segmentations acquired by constrained volume growing from manually placed seed points.
3.1 Cohort 1: Evaluation of seed point generator
At a threshold on the posterior probability of 0.9, the model with the consensus voting localized 95% of the lesions for the independent test set at a cost of 0.9 false positives per patient. Seed point generation took approximately 0.7 second per patient.
3.2 Cohort 1: Inspection of false negatives and false positives
We inspected the false negatives and false positives of the model after the consensus voting. The primary tumor was missed in one of the patients from the independent test set, this tumor was found by just one of the three network initializations. The false positives were focally enhancing regions — potentially indicative of benign disease (N = 4 patients), part of an intramammary blood vessel (N = 1 patient), and seemingly healthy enhancing parenchyma (N = 2 patients).
3.3 Cohort 2: Response to neoadjuvant chemotherapy
The constrained volume growing after consensus voting showed a median decrease in tumor volume of 70% (interquartile range: 50% – 77%), comparable to the reference standard (70%, interquartile range: 23% – 76%) (Figure 3). Inspection of the segmentations suggests that the method segments intramammary blood vessels more often in the patients receiving NAC (3/10, 30%) than in the patients with early breast cancer (1/20, 5%). Seed point generation took approximately 1.4 second per patient.
We presented a fully automated method for response monitoring to neoadjuvant chemotherapy (NAC). The method showed tumor shrinkage comparable to our reference standard. Hence, these preliminary results suggests that this method is feasible in a response monitoring workflow.
The method replaced a manual step in our workflow for response monitoring to NAC. Thus far, seed points were manually placed, after which a constrained volume growing algorithm segmented the tumor. In this study, we automatically calculated these seed points in roughly a second per patient. The primary tumor was located in 95% of the patients at the cost of 0.9 false positives per patient — comparable to literature . The seed point generator was applied as input for the constrained volume growing in patients receiving NAC. The change in tumor volume between the MRI before NAC and the MRI during NAC (median decrease 70%, interquartile range: 50% – 77%) was comparable to our reference standard (median decrease 70%, interquartile range: 23% – 76%).
False positives mainly occur in the intramammary blood vessels. Therefore, future work could investigate suppressing the blood vessels, either by preprocessing[22, 23], or by segmenting blood vessels as a separate class. That way, the network can potentially better distinguish them.
Regardless of these shortcomings, the method does perform comparable to our reference standard. Note that the method is trained on a different patient population, with MR images from a different vendor (Philips instead of Siemens), and differences in MR acquisition (3T vs 1.5T, slightly different voxel size, different repetition time, echo time, and flip angle). Hence, the method appears to be relatively agnostic to these patient and imaging differences.
In conclusion, response monitoring of breast cancer to neoadjuvant chemotherapy on dynamic contrast-enhanced MRI using convolutional neural network-generated seed points and constrained volume growing is feasible.
5 New or Breakthrough Work
A fast and automatic seed point generator was developed, fully automating a well-validated semi-automatic workflow for response monitoring of breast cancer to neoadjuvant chemotherapy.
This work is partially funded by the Dutch Cancer Society (KWF), grant number 10755, and by the Center for Translational Molecular Medicine (CTMM) grants Breast CaRe and CHOICE. We thank NVIDIA for the generous donation of a GPU.
-  Fisher, B., Brown, A., Mamounas, E., Wieand, S., Robidoux, A., Margolese, R. G., Cruz, A. B., Fisher, E. R., Wickerham, D. L., Wolmark, N., DeCillis, A., Hoehn, J. L., Lees, A. W., and Dimitrov, N. V., “Effect of preoperative chemotherapy on local-regional disease in women with operable breast cancer: findings from National Surgical Adjuvant Breast and Bowel Project B-18,” J. Clin. Oncol. 15(7), 2483–2493 (1997).
-  Mauri, D., Pavlidis, N., and Ioannidis, J. P. A., “Neoadjuvant Versus Adjuvant Systemic Treatment in Breast Cancer: A Meta-Analysis,” JNCI J. Natl. Cancer Inst. 97(3), 188–194 (2005).
-  Mieog, J. S. D., Van der Hage, J. A., and Van de Velde, C. J. H., “Neoadjuvant chemotherapy for operable breast cancer.,” Br. J. Surg. 94(10), 1189–200 (2007).
-  Steenbruggen, T. G., Van Ramshorst, M. S., Kok, M., Linn, S. C., Smorenburg, C. H., and Sonke, G. S., “Neoadjuvant Therapy for Breast Cancer: Established Concepts and Emerging Strategies.,” Drugs 77(12), 1313–1336 (2017).
-  Hylton, N. M., Blume, J. D., Bernreuter, W. K., Pisano, E. D., Rosen, M. A., Morris, E. A., Weatherall, P. T., Lehman, C. D., Newstead, G. M., Polin, S., Marques, H. S., Esserman, L. J., and Schnall, M. D., “Locally Advanced Breast Cancer: MR Imaging for Prediction of Response to Neoadjuvant Chemotherapy—Results from ACRIN 6657/I-SPY TRIAL,” Radiology 263(3), 663–672 (2012).
-  Gilhuijs, K. G. A., Deurloo, E. E., Muller, S. H., Peterse, J. L., and Schultze Kool, L. J., “Breast MR Imaging in Women at Increased Lifetime Risk of Breast Cancer: Clinical System for Computerized Assessment of Breast Lesions—Initial Results,” Radiology 225(3), 907–916 (2002).
-  Alderliesten, T., Schlief, A., Peterse, J., Loo, C., Teertstra, H., Muller, S., and Gilhuijs, K., “Validation of semiautomatic measurement of the extent of breast tumors using contrast-enhanced magnetic resonance imaging.,” Invest. Radiol. 42(1), 42–9 (2007).
-  Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., Van der Laak, J. A., Van Ginneken, B., and Sánchez, C. I., “A survey on deep learning in medical image analysis,” Med. Image Anal. 42, 60–88 (2017).
-  Dalmış, M. U., Litjens, G., Holland, K., Setio, A., Mann, R., Karssemeijer, N., and Gubern-Mérida, A., “Using deep learning to segment breast and fibroglandular tissue in MRI volumes,” Med. Phys. 44(2), 533–546 (2017).
-  Moeskops, P., Wolterink, J. M., Van der Velden, B. H. M., Gilhuijs, K. G. A., Leiner, T., Viergever, M. A., and Išgum, I., “Deep Learning for Multi-task Medical Image Segmentation in Multiple Modalities,” in [Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) ], 9901 LNCS, 478–486, Springer, Cham (2016).
-  Amit, G., Ben-Ari, R., Hadad, O., Monovich, E., Granot, N., and Hashoul, S., “Classification of breast MRI lesions using small-size training sets: comparison of deep learning approaches,” 10134, 101341H, International Society for Optics and Photonics (2017).
-  Elshof, L. E., Rutgers, E. J. T., Deurloo, E. E., Loo, C. E., Wesseling, J., Pengel, K. E., and Gilhuijs, K. G. A., “A practical approach to manage additional lesions at preoperative breast MRI in patients eligible for breast conserving therapy: Results,” Breast Cancer Res. Treat. 124(3), 707–715 (2010).
-  Loo, C. E., Straver, M. E., Rodenhuis, S., Muller, S. H., Wesseling, J., Vrancken Peeters, M.-J. T., and Gilhuijs, K. G., “Magnetic Resonance Imaging Response Monitoring of Breast Cancer During Neoadjuvant Chemotherapy: Relevance of Breast Cancer Subtype,” J. Clin. Oncol. 29(6), 660–666 (2011).
-  Dmitriev, I. D., Loo, C. E., Vogel, W. V., Pengel, K. E., and Gilhuijs, K. G. A., “Fully automated deformable registration of breast DCE-MRI and PET/CT.,” Phys. Med. Biol. 58(4), 1221–33 (2013).
-  Van der Velden, B. H. M., Dmitriev, I., Loo, C. E., Pijnappel, R. M., and Gilhuijs, K. G. A., “Association between Parenchymal Enhancement of the Contralateral Breast in Dynamic Contrast-enhanced MR Imaging and Outcome of Patients with Unilateral Invasive Breast Cancer,” Radiology 276(3), 675–85 (2015).
-  Tustison, N. J., Avants, B. B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., and Gee, J. C., “N4ITK: Improved N3 bias correction,” IEEE Trans. Med. Imaging 29(6), 1310–1320 (2010).
-  Gilhuijs, K. G., Giger, M. L., and Bick, U., “Computerized analysis of breast lesions in three dimensions using dynamic magnetic-resonance imaging.,” Med. Phys. 25(9), 1647–54 (1998).
-  Morris, E., Comstock, C., and Lee, C., [ACR BI-RADS Magnetic Resonance Imaging ], American College of Radiology, Reston, VA (2013).
-  Yu, F. and Koltun, V., “Multi-Scale Context Aggregation by Dilated Convolutions,” in [ICLR ], (2016).
-  Wolterink, J. M., Leiner, T., Viergever, M. A., and Išgum, I., “Dilated convolutional neural networks for cardiovascular MR segmentation in congenital heart disease,” in [Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) ], 10129 LNCS, 95–102, Springer, Cham (2017).
-  Gubern-Mérida, A., Martí, R., Melendez, J., Hauth, J. L., Mann, R. M., Karssemeijer, N., and Platel, B., “Automated localization of breast cancer in DCE-MRI,” Med. Image Anal. 20(1), 265–274 (2015).
-  Vignati, A., Giannini, V., Carbonaro, L. A., Bertotto, I., Martincich, L., Sardanelli, F., and Regge, D., “A new algorithm for automatic vascular mapping of DCE-MRI of the breast: Clinical application of a potential new biomarker.,” Comput. Methods Programs Biomed. 117(3), 482–8 (2014).
-  Van der Velden, B. H. M., Schmitz, A. M. T., Loo, C. E., and Gilhuijs, K. G. A., “Abstract P4-02-07: Association between computer-derived features of the ipsilateral breast on DCE-MRI and the 70-gene signature in patients with invasive breast cancer,” Cancer Res. 76(4) (2016).