1 Introduction
According to the American Cancer Society, breast cancer is the most frequently diagnosed solid cancer and the second leading cause of cancer death among U.S. women [1]. Mammogram screening has been demonstrated to be an effective way for early detection and diagnosis, which can significantly decrease breast cancer mortality [15]. Traditional mammogram classification requires extra annotations such as bounding box for detection or mask ground truth for segmentation [17, 5, 11]. Other work have employed different deep networks to detect ROIs and obtain mass boundaries in different stages [6]. However, these methods require handcrafted features to complement the system [12], and training data to be annotated with bounding boxes and segmentation ground truths which require expert domain knowledge and costly effort to obtain. In addition, multistage training cannot fully explore the power of deep networks.
Due to the high cost of annotation, we intend to perform classification based on a raw whole mammogram. Each patch of a mammogram can be treated as an instance and a whole mammogram is treated as a bag of instances. The whole mammogram classification problem can then be thought of as a standard MIL problem. Due to the great representation power of deep features
[9, 21, 19, 20], combining MIL with deep neural networks is an emerging topic. Yan et al. used a deep MIL to find discriminative patches for body part recognition
[18]. Patch based CNN added a new layer after the last layer of deep MIL to learn the fusion model for multiinstance predictions [10]. Shen et al. employed two stage training to learn the deep multiinstance networks for predetected lung nodule classification [16]. The above approaches used max pooling to model the general multiinstance assumption which only considers the patch of max probability. In this paper, more effective taskrelated deep multiinstance models with endtoend training are explored for whole mammogram classification. We investigate three different schemes, i.e., max pooling, label assignment, and sparsity, to perform deep MIL for the whole mammogram classification task.
The framework for our proposed endtoend trained deep MIL for mammogram classification is shown in Fig. 1. To fully explore the power of deep MIL, we convert the traditional MIL assumption into a label assignment problem. As a mass typically composes only 2% of a whole mammogram (see Fig. 2), we further propose sparse deep MIL. The proposed deep multiinstance networks are shown to provide robust performance for whole mammogram classification on the INbreast dataset [14].
2 Deep MIL for Whole Mammogram Mass Classification
Unlike other deep multiinstance networks [18, 10], we use a CNN to efficiently obtain features of all patches (instances) at the same time. Given an image , we obtain a much smaller feature map of multichannels after multiple convolutional layers and max pooling layers. The represents deep CNN features for a patch in , where represents the pixel row and column index respectively.
The goal of our work is to predict whether a whole mammogram contains a malignant mass (BIRADS as positive) or not, which is a standard binary classification problem. We add a logistic regression with weights shared across all the pixel positions following
, and an elementwise sigmoid activation function is applied to the output. To clarify it, the malignant probability of feature space’s pixel
is(1) 
where is the weights in logistic regression, is the bias, and
is the inner product of the two vectors
and . The and are shared for different pixel positions . We can combine into a matrix of range denoting the probabilities of patches being malignant masses. The can be flattened into a onedimensional vector as corresponding to flattened patches , where is the number of patches.2.1 Max Poolingbased Multiinstance Learning
The general multiinstance assumption is that if there exists an instance that is positive, the bag is positive [7]. The bag is negative if and only if all instances are negative. For whole mammogram classification, the equivalent scenario is that if there exists a malignant mass, the mammogram
should be classified as positive. Likewise, negative mammogram
should not have any malignant masses. If we treat each patch of as an instance, the whole mammogram classification is a standard multiinstance task.For negative mammograms, we expect all the to be close to 0. For positive mammograms, at least one should be close to 1. Thus, it is natural to use the maximum component of as the malignant probability of the mammogram
(2) 
where is the weights in deep networks.
If we sort first in descending order as illustrated in Fig. 1, the malignant probability of the whole mammogram is the first element of ranked as
(3)  
where is descending ranked . The cross entropybased cost function can be defined as
(4) 
where is the total number of mammograms, is the true label of malignancy for mammogram , and is the regularizer that controls model complexity.
One disadvantage of max poolingbased MIL is that it only considers the patch , and does not exploit information from other patches. A more powerful framework should add taskrelated priori, such as sparsity of mass in whole mammogram, into the general multiinstance assumption and explore more patches for training.
2.2 Label Assignmentbased Multiinstance Learning
For the conventional classification tasks, we assign a label to each data point. In the MIL scheme, if we consider each instance (patch) as a data point for classification, we can convert the multiinstance learning problem into a label assignment problem.
After we rank the malignant probabilities for all the instances (patches) in a whole mammogram using the first equation in Eq. 3, the first few should be consistent with the label of whole mammogram as previously mentioned, while the remaining patches (instances) should be negative. Instead of adopting the general MIL assumption that only considers the (patch of malignant probability ), we assume that 1) patches of the first largest malignant probabilities should be assigned with the same class label as that of whole mammogram, and 2) the rest patches should be labeled as negative in the label assignmentbased MIL.
After the ranking/sorting layer using the first equation in Eq. 3, we can obtain the malignant probability for each patch
(5) 
The cross entropy loss function of the label assignmentbased MIL can be defined
(6)  
One advantage of the label assignmentbased MIL is that it explores all the patches to train the model. Essentially it acts a kind of data augmentation which is an effective technique to train deep networks when the training data is scarce. From the sparsity perspective, the optimization problem of label assignmentbased MIL is exactly a sparse problem for the positive data points, where we expect being 1 and
being 0. The disadvantage of label assignmentbased MIL is that it is hard to estimate the hyperparameter
. Thus, a relaxed assumption for the MIL or an adaptive way to estimate the hyperparameter is preferred.2.3 Sparse Multiinstance Learning
From the mass distribution, the mass typically comprises about 2% of the whole mammogram on average (Fig. 2), which means the mass region is quite sparse in the whole mammogram. It is straightforward to convert the mass sparsity to the malignant mass sparsity, which implies that is sparse in the whole mammogram classification problem. The sparsity constraint means we expect the malignant probability of part patches being 0 or close to 0, which is equivalent to the second assumption in the label assignmentbased MIL. Analogously, we expect to be indicative of the true label of mammogram .
After the above discussion, the loss function of sparse MIL problem can be defined
(7) 
where can be calculated in Eq. 3, for mammogram , denotes the norm, is the sparsity factor, which is a tradeoff between the sparsity assumption and the importance of patch .
From the discussion of label assignmentbased MIL, this learning is a kind of exact sparse problem which can be converted to constrain. One advantage of sparse MIL over label assignmentbased MIL is that it does not require assign label for each patch which is hard to do for patches where probabilities are not too large or small. The sparse MIL considers the overall statistical property of
Another advantage of sparse MIL is that, it has different weights for general MIL assumption (the first part loss) and label distribution within mammogram (the second part loss), which can be considered as a tradeoff between max poolingbased MIL (slack assumption) and label assignmentbased MIL (hard assumption).
3 Experiments
We validate the proposed models on the most frequently used mammographic mass classification dataset, INbreast dataset [14], as the mammograms in other datasets, such as DDSM dataset [4], are of low quality. The INbreast dataset contains 410 mammograms of which 100 containing malignant masses. These 100 mammograms with malignant masses are defined as positive. For fair comparison, we also use 5fold cross validation to evaluate model performance as [6]. For each testing fold, we use three folds for training, and one fold for validation to tune hyperparameters. The performance is reported as the average of five testing results obtained from crossvalidation.
We employ techniques to augment our data. For each training epoch, we randomly flip the mammograms horizontally, shift within 0.1 proportion of mammograms horizontally and vertically, rotate within 45 degree, and set
square box as 0. In experiments, the data augmentation is essential for us to train the deep networks.For the CNN network structure, we use AlexNet and remove the fully connected layers [13]. Through CNN, the mammogram of size becomes 256 feature maps. Then we use steps in Sec. 2
to do MIL. Here we employ weights pretrained on the ImageNet due to the scarce of data. We use Adam optimization with learning rate
for training models [2]. The for max poolingbased and label assignmentbased MIL are . The and for sparse MIL are and respectively. For the label assignmentbased MIL.We firstly compare our methods to previous models validated on DDSM dataset and INbreast dataset in Table 1. Previous handcrafted featurebased methods require manually annotated detection bounding box or segmentation ground truth even in test denoting as manual [3, 17, 8]. The feat. denotes requiring handcrafted features. Pretrained CNN uses two CNNs to detect the mass region and segment the mass, followed by a third CNN to do mass classification on the detected ROI region, which requires handcrafted features to pretrain the network and needs multistages training[6]
. Pretrained CNN+Random Forest further employs random forest and obtained 7% improvement. These methods are either manually or need handcrafted features or multistages training, while our methods are totally automated, do not require handcrafted features or extra annotations even on training set, and can be trained in an endtoend manner.


Methodology  Dataset  Setup  Accu.  AUC  



DDSM  Manual+feat.  0.87  N/A  

DDSM  Manual+feat.  0.81  N/A  

INbr.  Manual+feat.  0.89  N/A  

INbr.  Auto.+feat.  0.84  0.69  

INbr.  Auto.+feat.  0.76  


AlexNet  INbr.  Auto.  0.81  0.79  

INbr.  Auto.  0.85  0.83  

INbr.  Auto.  0.86  0.84  

INbr.  Auto.  0.90  

The max poolingbased deep MIL obtains better performance than the pretrained CNN using 3 different CNNs and detection/segmentation annotation in the training set. This shows the superiority of our endtoend trained deep MIL for whole mammogram classification. According to the accuracy metric, the sparse deep MIL is better than the label assignmentbased MIL, which is better than the max poolingbased MIL. This result is consistent with previous discussion that the sparsity assumption benefited from not having hard constraints of the label assignment assumption, which employs all the patches and is more efficient than max pooling assumption. Our sparse deep MIL achieves competitive accuracy to random forestbased pretrained CNN, while much higher AUC than previous work, which shows our method is more robust. The main reasons for the robust results using our models are as follows. Firstly, data augmentation is an important technique to increase scarce training datasets and proves useful here. Secondly, the transfer learning that employs the pretrained weights from ImageNet is effective for the INBreast dataset. Thirdly, our models fully explore all the patches to train our deep networks thereby eliminating any possibility of overlooking malignant patches by only considering a subset of patches. This is a distinct advantage over previous networks employing several stages.
To further understand our deep MIL, we visualize the responses of logistic regression layer for four mammograms on test set, which represents the malignant probability of each patch, in Fig. 3. We can see the deep MIL learns not only the prediction of whole mammogram, but also the prediction of malignant patches within the whole mammogram. Our models are able to learn the mass region of the whole mammogram without any explicit bounding box or segmentation ground truth annotation of training data. The max poolingbased deep multiinstance network misses some malignant patches in (a), (c) and (d). The possible reason is that it only considers the patch of max malignant probability in training and the model is not well learned for all patches. The label assignmentbased deep MIL misclassifies some patches in (d). The possible reason is that the model sets a constant for all the mammograms, which causes some misclassifications for small masses. One of the potential applications of our work is that these deep MIL networks could be used to do weak mass annotation automatically, which provides evidence for the diagnosis.
4 Conclusion
In this paper, we propose endtoend trained deep MIL for whole mammogram classification. Different from previous work using segmentation or detection annotations, we conduct mass classification based on whole mammogram directly. We convert the general MIL assumption to label assignment problem after ranking. Due to the sparsity of masses, sparse MIL is used for whole mammogram classification. Experimental results demonstrate more robust performance even without detection or segmentation annotation in the training.
In future work, we plan to extend the current work by: 1) incorporating multiscale modeling such as spatial pyramid to further improve whole mammogram classification, 2) employing the deep MIL to do annotation or provide potential malignant patches to assist diagnoses, and 3) applying to large datasets if the big dataset is available.
References
 [1] American cancer society. what are the key statistics about breast cancer?
 [2] Ba, J., Kingma, D.: Adam: A method for stochastic optimization. ICLR (2015)
 [3] Ball, J.E., Bruce, L.M.: Digital mammographic computer aided diagnosis (cad) using adaptive level set segmentation. In: EMBS (2007)
 [4] Bowyer, K., Kopans, D., Kegelmeyer, W., et al.: The digital database for screening mammography. In: IWDM (1996)

[5]
Carneiro, G., Nascimento, J., Bradley, A.P.: Unregistered multiview mammogram analysis with pretrained deep learning models. In: MICCAI. pp. 652–660. Springer (2015)
 [6] Dhungel, N., Carneiro, G., Bradley, A.P.: The automated learning of deep features for breast mass classification from mammograms. In: MICCAI. pp. 106–114. Springer (2016)

[7]
Dietterich, T.G., Lathrop, R.H., LozanoPérez, T.: Solving the multiple instance problem with axisparallel rectangles. Artificial intelligence 89(1), 31–71 (1997)
 [8] Domingues, I., Sales, E., Cardoso, J., Pereira, W.: Inbreastdatabase masses characterization. XXIII CBEB (2012)
 [9] Greenspan, H., van Ginneken, B., Summers, R.M.: Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging 35(5), 1153–1159 (2016)
 [10] Hou, L., Samaras, D., Kurc, T.M., et al.: Patchbased convolutional neural network for whole slide tissue image classification. arXiv:1504.07947 (2015)
 [11] Jiao, Z., Gao, X., Wang, Y., Li, J.: A deep feature based framework for breast masses classification. Neurocomputing (2016)
 [12] Kooi, T., Litjens, G., van Ginneken, B., et al.: Large scale deep learning for computer aided detection of mammographic lesions. Medical image analysis 35, 303–312 (2017)
 [13] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. pp. 1097–1105 (2012)
 [14] Moreira, I.C., Amaral, I., Domingues, I., et al.: Inbreast: toward a fullfield digital mammographic database. Academic radiology (2012)
 [15] Oeffinger, K.C., Fontham, E.T., Etzioni, R., et al.: Breast cancer screening for women at average risk: 2015 guideline update from the american cancer society. Jama (2015)
 [16] Shen, W., Zhou, M., Yang, F., et al.: Learning from experts: Developing transferable deep features for patientlevel lung cancer prediction. In: MICCAI (2016)
 [17] Varela, C., Timp, S., Karssemeijer, N.: Use of border information in the classification of mammographic masses. Physics in Medicine and Biology 51(2), 425 (2006)
 [18] Yan, Z., Zhan, Y., Peng, Z., et al.: Multiinstance deep learning: Discover discriminative local anatomies for bodypart recognition. IEEE transactions on medical imaging 35(5)
 [19] Zhu, W., Lan, C., Xing, J., et al.: Cooccurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In: AAAI (2016)
 [20] Zhu, W., Miao, J., Qing, L., Huang, G.B.: Hierarchical extreme learning machine for unsupervised representation learning. In: IJCNN. pp. 1–8. IEEE (2015)
 [21] Zhu, W., Xie, X.: Adversarial deep structural networks for mammographic mass segmentation. arXiv:1612.05970 (2016)
Comments
There are no comments yet.