According to the American Cancer Society, breast cancer is the most frequently diagnosed solid cancer and the second leading cause of cancer death among U.S. women . Mammogram screening has been demonstrated to be an effective way for early detection and diagnosis, which can significantly decrease breast cancer mortality . Traditional mammogram classification requires extra annotations such as bounding box for detection or mask ground truth for segmentation [17, 5, 11]. Other work have employed different deep networks to detect ROIs and obtain mass boundaries in different stages . However, these methods require hand-crafted features to complement the system , and training data to be annotated with bounding boxes and segmentation ground truths which require expert domain knowledge and costly effort to obtain. In addition, multi-stage training cannot fully explore the power of deep networks.
Due to the high cost of annotation, we intend to perform classification based on a raw whole mammogram. Each patch of a mammogram can be treated as an instance and a whole mammogram is treated as a bag of instances. The whole mammogram classification problem can then be thought of as a standard MIL problem. Due to the great representation power of deep features[9, 21, 19, 20]
, combining MIL with deep neural networks is an emerging topic. Yan et al. used a deep MIL to find discriminative patches for body part recognition. Patch based CNN added a new layer after the last layer of deep MIL to learn the fusion model for multi-instance predictions . Shen et al. employed two stage training to learn the deep multi-instance networks for pre-detected lung nodule classification 
. The above approaches used max pooling to model the general multi-instance assumption which only considers the patch of max probability. In this paper, more effective task-related deep multi-instance models with end-to-end training are explored for whole mammogram classification. We investigate three different schemes, i.e., max pooling, label assignment, and sparsity, to perform deep MIL for the whole mammogram classification task.
The framework for our proposed end-to-end trained deep MIL for mammogram classification is shown in Fig. 1. To fully explore the power of deep MIL, we convert the traditional MIL assumption into a label assignment problem. As a mass typically composes only 2% of a whole mammogram (see Fig. 2), we further propose sparse deep MIL. The proposed deep multi-instance networks are shown to provide robust performance for whole mammogram classification on the INbreast dataset .
2 Deep MIL for Whole Mammogram Mass Classification
Unlike other deep multi-instance networks [18, 10], we use a CNN to efficiently obtain features of all patches (instances) at the same time. Given an image , we obtain a much smaller feature map of multi-channels after multiple convolutional layers and max pooling layers. The represents deep CNN features for a patch in , where represents the pixel row and column index respectively.
The goal of our work is to predict whether a whole mammogram contains a malignant mass (BI-RADS as positive) or not, which is a standard binary classification problem. We add a logistic regression with weights shared across all the pixel positions following
, and an element-wise sigmoid activation function is applied to the output. To clarify it, the malignant probability of feature space’s pixelis
where is the weights in logistic regression, is the bias, and
is the inner product of the two vectorsand . The and are shared for different pixel positions . We can combine into a matrix of range denoting the probabilities of patches being malignant masses. The can be flattened into a one-dimensional vector as corresponding to flattened patches , where is the number of patches.
2.1 Max Pooling-based Multi-instance Learning
The general multi-instance assumption is that if there exists an instance that is positive, the bag is positive . The bag is negative if and only if all instances are negative. For whole mammogram classification, the equivalent scenario is that if there exists a malignant mass, the mammogram
should be classified as positive. Likewise, negative mammogramshould not have any malignant masses. If we treat each patch of as an instance, the whole mammogram classification is a standard multi-instance task.
For negative mammograms, we expect all the to be close to 0. For positive mammograms, at least one should be close to 1. Thus, it is natural to use the maximum component of as the malignant probability of the mammogram
where is the weights in deep networks.
If we sort first in descending order as illustrated in Fig. 1, the malignant probability of the whole mammogram is the first element of ranked as
where is descending ranked . The cross entropy-based cost function can be defined as
where is the total number of mammograms, is the true label of malignancy for mammogram , and is the regularizer that controls model complexity.
One disadvantage of max pooling-based MIL is that it only considers the patch , and does not exploit information from other patches. A more powerful framework should add task-related priori, such as sparsity of mass in whole mammogram, into the general multi-instance assumption and explore more patches for training.
2.2 Label Assignment-based Multi-instance Learning
For the conventional classification tasks, we assign a label to each data point. In the MIL scheme, if we consider each instance (patch) as a data point for classification, we can convert the multi-instance learning problem into a label assignment problem.
After we rank the malignant probabilities for all the instances (patches) in a whole mammogram using the first equation in Eq. 3, the first few should be consistent with the label of whole mammogram as previously mentioned, while the remaining patches (instances) should be negative. Instead of adopting the general MIL assumption that only considers the (patch of malignant probability ), we assume that 1) patches of the first largest malignant probabilities should be assigned with the same class label as that of whole mammogram, and 2) the rest patches should be labeled as negative in the label assignment-based MIL.
After the ranking/sorting layer using the first equation in Eq. 3, we can obtain the malignant probability for each patch
The cross entropy loss function of the label assignment-based MIL can be defined
One advantage of the label assignment-based MIL is that it explores all the patches to train the model. Essentially it acts a kind of data augmentation which is an effective technique to train deep networks when the training data is scarce. From the sparsity perspective, the optimization problem of label assignment-based MIL is exactly a -sparse problem for the positive data points, where we expect being 1 and
being 0. The disadvantage of label assignment-based MIL is that it is hard to estimate the hyper-parameter. Thus, a relaxed assumption for the MIL or an adaptive way to estimate the hyper-parameter is preferred.
2.3 Sparse Multi-instance Learning
From the mass distribution, the mass typically comprises about 2% of the whole mammogram on average (Fig. 2), which means the mass region is quite sparse in the whole mammogram. It is straightforward to convert the mass sparsity to the malignant mass sparsity, which implies that is sparse in the whole mammogram classification problem. The sparsity constraint means we expect the malignant probability of part patches being 0 or close to 0, which is equivalent to the second assumption in the label assignment-based MIL. Analogously, we expect to be indicative of the true label of mammogram .
After the above discussion, the loss function of sparse MIL problem can be defined
where can be calculated in Eq. 3, for mammogram , denotes the norm, is the sparsity factor, which is a trade-off between the sparsity assumption and the importance of patch .
From the discussion of label assignment-based MIL, this learning is a kind of exact -sparse problem which can be converted to constrain. One advantage of sparse MIL over label assignment-based MIL is that it does not require assign label for each patch which is hard to do for patches where probabilities are not too large or small. The sparse MIL considers the overall statistical property of
Another advantage of sparse MIL is that, it has different weights for general MIL assumption (the first part loss) and label distribution within mammogram (the second part loss), which can be considered as a trade-off between max pooling-based MIL (slack assumption) and label assignment-based MIL (hard assumption).
We validate the proposed models on the most frequently used mammographic mass classification dataset, INbreast dataset , as the mammograms in other datasets, such as DDSM dataset , are of low quality. The INbreast dataset contains 410 mammograms of which 100 containing malignant masses. These 100 mammograms with malignant masses are defined as positive. For fair comparison, we also use 5-fold cross validation to evaluate model performance as . For each testing fold, we use three folds for training, and one fold for validation to tune hyper-parameters. The performance is reported as the average of five testing results obtained from cross-validation.
We employ techniques to augment our data. For each training epoch, we randomly flip the mammograms horizontally, shift within 0.1 proportion of mammograms horizontally and vertically, rotate within 45 degree, and setsquare box as 0. In experiments, the data augmentation is essential for us to train the deep networks.
to do MIL. Here we employ weights pretrained on the ImageNet due to the scarce of data. We use Adam optimization with learning ratefor training models . The for max pooling-based and label assignment-based MIL are . The and for sparse MIL are and respectively. For the label assignment-based MIL.
We firstly compare our methods to previous models validated on DDSM dataset and INbreast dataset in Table 1. Previous hand-crafted feature-based methods require manually annotated detection bounding box or segmentation ground truth even in test denoting as manual [3, 17, 8]. The feat. denotes requiring hand-crafted features. Pretrained CNN uses two CNNs to detect the mass region and segment the mass, followed by a third CNN to do mass classification on the detected ROI region, which requires hand-crafted features to pretrain the network and needs multi-stages training
. Pretrained CNN+Random Forest further employs random forest and obtained 7% improvement. These methods are either manually or need hand-crafted features or multi-stages training, while our methods are totally automated, do not require hand-crafted features or extra annotations even on training set, and can be trained in an end-to-end manner.
The max pooling-based deep MIL obtains better performance than the pretrained CNN using 3 different CNNs and detection/segmentation annotation in the training set. This shows the superiority of our end-to-end trained deep MIL for whole mammogram classification. According to the accuracy metric, the sparse deep MIL is better than the label assignment-based MIL, which is better than the max pooling-based MIL. This result is consistent with previous discussion that the sparsity assumption benefited from not having hard constraints of the label assignment assumption, which employs all the patches and is more efficient than max pooling assumption. Our sparse deep MIL achieves competitive accuracy to random forest-based pretrained CNN, while much higher AUC than previous work, which shows our method is more robust. The main reasons for the robust results using our models are as follows. Firstly, data augmentation is an important technique to increase scarce training datasets and proves useful here. Secondly, the transfer learning that employs the pretrained weights from ImageNet is effective for the INBreast dataset. Thirdly, our models fully explore all the patches to train our deep networks thereby eliminating any possibility of overlooking malignant patches by only considering a subset of patches. This is a distinct advantage over previous networks employing several stages.
To further understand our deep MIL, we visualize the responses of logistic regression layer for four mammograms on test set, which represents the malignant probability of each patch, in Fig. 3. We can see the deep MIL learns not only the prediction of whole mammogram, but also the prediction of malignant patches within the whole mammogram. Our models are able to learn the mass region of the whole mammogram without any explicit bounding box or segmentation ground truth annotation of training data. The max pooling-based deep multi-instance network misses some malignant patches in (a), (c) and (d). The possible reason is that it only considers the patch of max malignant probability in training and the model is not well learned for all patches. The label assignment-based deep MIL mis-classifies some patches in (d). The possible reason is that the model sets a constant for all the mammograms, which causes some mis-classifications for small masses. One of the potential applications of our work is that these deep MIL networks could be used to do weak mass annotation automatically, which provides evidence for the diagnosis.
In this paper, we propose end-to-end trained deep MIL for whole mammogram classification. Different from previous work using segmentation or detection annotations, we conduct mass classification based on whole mammogram directly. We convert the general MIL assumption to label assignment problem after ranking. Due to the sparsity of masses, sparse MIL is used for whole mammogram classification. Experimental results demonstrate more robust performance even without detection or segmentation annotation in the training.
In future work, we plan to extend the current work by: 1) incorporating multi-scale modeling such as spatial pyramid to further improve whole mammogram classification, 2) employing the deep MIL to do annotation or provide potential malignant patches to assist diagnoses, and 3) applying to large datasets if the big dataset is available.
-  American cancer society. what are the key statistics about breast cancer?
-  Ba, J., Kingma, D.: Adam: A method for stochastic optimization. ICLR (2015)
-  Ball, J.E., Bruce, L.M.: Digital mammographic computer aided diagnosis (cad) using adaptive level set segmentation. In: EMBS (2007)
-  Bowyer, K., Kopans, D., Kegelmeyer, W., et al.: The digital database for screening mammography. In: IWDM (1996)
Carneiro, G., Nascimento, J., Bradley, A.P.: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: MICCAI. pp. 652–660. Springer (2015)
-  Dhungel, N., Carneiro, G., Bradley, A.P.: The automated learning of deep features for breast mass classification from mammograms. In: MICCAI. pp. 106–114. Springer (2016)
Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence 89(1), 31–71 (1997)
-  Domingues, I., Sales, E., Cardoso, J., Pereira, W.: Inbreast-database masses characterization. XXIII CBEB (2012)
-  Greenspan, H., van Ginneken, B., Summers, R.M.: Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging 35(5), 1153–1159 (2016)
-  Hou, L., Samaras, D., Kurc, T.M., et al.: Patch-based convolutional neural network for whole slide tissue image classification. arXiv:1504.07947 (2015)
-  Jiao, Z., Gao, X., Wang, Y., Li, J.: A deep feature based framework for breast masses classification. Neurocomputing (2016)
-  Kooi, T., Litjens, G., van Ginneken, B., et al.: Large scale deep learning for computer aided detection of mammographic lesions. Medical image analysis 35, 303–312 (2017)
-  Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. pp. 1097–1105 (2012)
-  Moreira, I.C., Amaral, I., Domingues, I., et al.: Inbreast: toward a full-field digital mammographic database. Academic radiology (2012)
-  Oeffinger, K.C., Fontham, E.T., Etzioni, R., et al.: Breast cancer screening for women at average risk: 2015 guideline update from the american cancer society. Jama (2015)
-  Shen, W., Zhou, M., Yang, F., et al.: Learning from experts: Developing transferable deep features for patient-level lung cancer prediction. In: MICCAI (2016)
-  Varela, C., Timp, S., Karssemeijer, N.: Use of border information in the classification of mammographic masses. Physics in Medicine and Biology 51(2), 425 (2006)
-  Yan, Z., Zhan, Y., Peng, Z., et al.: Multi-instance deep learning: Discover discriminative local anatomies for bodypart recognition. IEEE transactions on medical imaging 35(5)
-  Zhu, W., Lan, C., Xing, J., et al.: Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In: AAAI (2016)
-  Zhu, W., Miao, J., Qing, L., Huang, G.B.: Hierarchical extreme learning machine for unsupervised representation learning. In: IJCNN. pp. 1–8. IEEE (2015)
-  Zhu, W., Xie, X.: Adversarial deep structural networks for mammographic mass segmentation. arXiv:1612.05970 (2016)