Two-Stage Convolutional Neural Network for Breast Cancer Histology Image Classification

03/11/2018 ∙ by Kamyar Nazeri, et al. ∙ University of Ontario Institute of Technology 0

This paper explores the problem of breast tissue classification of microscopy images. Based on the predominant cancer type the goal is to classify images into four categories of normal, benign, in situ carcinoma, and invasive carcinoma. Given a suitable training dataset, we utilize deep learning techniques to address the classification problem. Due to the large size of each image in the training dataset, we propose a patch-based technique which consists of two consecutive convolutional neural networks. The first "patch-wise" network extracts features of image patches while the second "image-wise" network performs classification of the whole image. The first network is aimed at extracting local information while the second network obtains global information of an input image. We trained the networks using the ICIAR 2018 grand challenge on BreAst Cancer Histology (BACH) dataset. The proposed method yields 95 previously reported 77 available at



There are no comments yet.


page 1

page 2

page 3

page 4

Code Repositories


Two-Stage Convolutional Neural Network for Breast Cancer Histology Image Classification. ICIAR 2018 Grand Challenge on BreAst Cancer Histology images (BACH)

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Breast cancer is one of the leading causes of cancer-related death in women around the world [1]. According to Canadian Cancer Society, over women were diagnosed with breast cancer in Canada in 2017 which represents of all new cancer cases in women. In the same year, more than women in Canada lost their lives due to breast cancer which represents of all cancer deaths in women. It is evident that early diagnosis can significantly increase treatment success. Breast cancer symptoms and signs are varied and diagnosis includes physical exam, mammography, ultrasound testing, and biopsy. Biopsy is generally performed after detection of some abnormality using mammography and ultrasound. In biopsy, a sample of tissue is surgically removed to be analyzed. This can indicate which cells are cancerous, and if so which type of cancer these are associated to. Microscopy imaging data of biopsy samples are large in size and complex in nature. Therefore, pathologists face a substantial workload increase for histopathological cancer diagnosis. In recent years, the development of computer aid diagnosis (CAD) systems have helped reducing this workload. Digital pathology continues to gain momentum worldwide for diagnostic purposes [2]

. Recently, deep learning techniques have emerged to address many problems in the field of medical image processing. We propose a classification scheme for breast cancer tissue image classification based on deep convolutional neural networks (CNN). Convolutional networks are considered state of the art technique for classification problems when the input is high-dimensional data such as images. These networks “learn” to extract local features from images and classify the input according to the extracted features. Size of microscopy images are very large and due to hardware barriers, several patch-based CNN methods have been proposed in the literature

[3, 4, 5]

to process the input image as a set of smaller patches. In these models, each image is divided into smaller patches and each patch is classified with a “patch-wise” classifier network and assigned to a label. To classify at the whole image level, the patch-wise network is followed by another classifier that receives output labels from the first network as input and generates label scores. These techniques achieve high accuracy with high confidence on image patches, however, they fail to capture global attributes of the image: Once all image patches are labelled, the spatial information is ignored and any possible feature that is shared between patches is lost. We propose a novel two-stage convolutional neural network pipeline in a patch-wise fashion that is designed to utilize both local and global information of the input. The proposed method does not require a large memory footprint of the end-to-end training. In this scheme, the sole purpose of the patch-wise network is to extract spatially smaller feature maps from each patch. Once trained, this network is then used to extract the most salient feature maps from all patches in an image based on their local information. These feature maps are stacked together to form a spatially smaller 3D input for the “image-wise” network. This network is trained to classify images based on local features extracted from image and global information shared between different patches. We trained our network using the ICIAR 2018 grand challenge on BreAst Cancer Histology (BACH) dataset

[6] containing Hematoxylin and Eosin (H&E) stained breast histology microscopy images. Our model has achieved 95% accuracy on the validation set, outperforming [3] in terms of classification accuracy. 111Our code and pre-trained weights are available at

2 Related Works

Due to the importance of detection and classification of breast cancer in microscopic tissue images, many new methods have emerged in recent years [3, 4, 5]. Computer aided diagnosis (CAD) systems appear to become fast and inexpensive alternatives to second opinion methods. Recently, deep learning techniques have made a huge impact in various problems including medical image processing. In the past few years, several works aimed at breast cancer detection and classification using CNNs have been published [7, 8, 9, 10, 11, 5, 12]. Although the aim of all of these works are very similar, each work considers a specific type of problem. For example, [7, 8, 9] are proposing a two class (malignant and benign) classifier. Other works in [10, 11] consider more complex 3-class classification (normal, in situ carcinoma, and invasive carcinoma). Finally, [5, 12] develop a segmentation scheme for breast cancer. Our work is similar to the work of Araújo et al. [3]

in nature. To the best of our knowledge they were the first team to consider a four class classifier for breast tissue images. They developed a CNN followed by a support vector machines (SVM) classifier. In their technique, first the original image is divided into twelve contiguous non-overlapping patches. The patch class probability is computed using the patch-wise trained CNN and CNN+SVM classifiers. Finally, the image-wise classification is obtained using three different patch probability fusion methods. These three methods namely include “majority voting”, “maximum probability”, and “sum of probabilities”

[3]. Another realted work which is in field of breast cancer detection is the work of Cruz-Roa et al. [cruz2014automatic.Thisworkisconsideringthetaskofdetectinginvasiveductalcarcinomatissueregionsinwholeslideimages.Theirapproachconsistofthreesteps, firsteachwholeslideimagewillbedividedintononoverlapping$100×100$pixelspatches, thebackgroundandfat-containingpatcheswillberemoved.Innextsteptheytraineda3layerconvolutionalneuralnetwork.Atthelaststep, aprobabilitymapforeachwholeslideimagewillbebuildbyattachingthenetworkoutputforeachpatch.\par\end{document}

3 Methods

Given a high resolution () histology image, our goal is to classify the image into four classes: normal tissue, benign tissue, in situ carcinoma and invasive carcinoma.

3.1 Patch-Based Method with CNN

The high resolution nature of the images in our dataset and the need to extract relevant discriminatory features from them impose extra limitations in implementing a regular feed forward convolutional network. Training a CNN on high resolution image requires either a very large memory footprint, which is not available in most cases, or to progressively reduce the spatial size of the image such that the downsampled version could be stored in the memory. However, downsampling an image increases the risk of losing discriminative features such as nuclei information and their densities to correctly classify carcinoma versus non-carcinoma cells. Also, if trained on the large microscopy image, the network might learn to rely only on the most distinctive features and totally discard everything else. We follow the patch-wise CNN method proposed by [3, 4, 5] followed by an image-wise CNN that classifies histology images into four classes. Given a microscopy image, we extract fixed size patches by sliding a patch (window) of size

with stride of

over an image. This makes a total number of patches where and are image width and height respectively. In our experiments, we follow [3] and choose patch size of considering the amount of GPU memory available. We also choose a stride of , which results in overlapping image patches. We argue that allowing the overlap is essential for the patch-wise network to learn features shared between patches. The proper stride is chosen by considering the receptive field of both networks when they “work” together, as explained later. An overview of our two-stage CNN is presented in Figure 1. Note that the labels in the training set are provided only for the whole image and individual patch labels are unknown, yet we train the patch-wise network using categorical cross-entropy loss based on the label of the corresponding microscopy image. This network works like an auto-encoder that learns to extract the most salient features of image patches. Once trained, we discard the classifier layer of this network and use the last convolutional layer to extract feature maps of size from any number of patches in an image, where is another hyper-parameter in our proposed system that controls the depth of output feature maps as explained later. To train the image-wise network, we no longer extract overlapping patches from the image: with stride of patches do not overlap and the total patches extracted from an image becomes . We found non-overlapping patches work slightly better in our validation set. We argue that it is because overlapping patches introduce redundant features for a single concept and as a result accuracy of the image-wise network will suffer. The extracted feature-maps from all 12 patches are concatenated together to form a spatially smaller 3D input of size for the image-wise network. This network is trained against image-level labels using categorical cross-entropy loss and learns to classify images based on local features extracted from image patches and global information shared between different patches.

Figure 1: An overview of the proposed workflow. A CNN is trained on image patches. Feature maps extracted from all image patches are stacked together and passed to a second CNN that learns to predict image-level labels.

Once both networks are trained, we use them jointly to infer image-level class predictions.

3.2 Network Architecture

Inspired by [13], we design our patch-wise CNN using a series of convolutional layers followed by a pooling layer with the number of channels being doubled after each downsampling. All convolutional layers are followed by batch normalization [14] and ReLU non-linearity [15]. We followed the guideline in [16] to implement a homogeneous fully convolutional network with occasional dimensionality reduction by using a stride of . In our tests, we found that kernel with stride of

worked better than conventional max-pooling layers in terms of performance. Instead of fully connected layers for the classification task, we use a

convolutional layer to obtain the spatial average of feature maps from the convolutional layer below it, as the confidence categories and the resulting vector is fed into the softmax layer [17]. We use this feature map later as an input to the image-wise CNN. To further control and experiment the effect of using spatial averaging layer, we introduce another hyper-parameter

that controls the depth of the output feature maps. Both batch normalization and global average pooling are

structural regularizers [17, 14] which natively prevent overfitting. As a result, we did not introduce any dropout or weight decay in our model. Overall, there are convolutional layers in the network with the input being downsampled times at layers , , and . Figure 2 illustrates the overall structure of the proposed patch-wise network.

Figure 2: Patch-wise network architecture

For the proposed image-wise network we follow a similar pattern. Series of convolutional layers are followed by a convolution with stride of

for downsampling. Each layer is followed by batch normalization and ReLU activation function. We use the same

convolutional layer as before to obtain the spatial average of activation maps before the classifier. The convolutional layers are followed by fully connected layers with a softmax classifier at the end. Unlike the patch-wise network, overfitting is a major problem for this network, as a result, we heavily regularize this network using dropout [18] with the rate of and use early stopping once the validation accuracy doesn’t improve to limit overfitting. Network architecture is shown in Figure 3.

Figure 3: Image-wise network architecture

The receptive field of the last convolutional layer with respect to the patch-wise network is [19]. In principle, this number has to be the maximum stride value we can choose in extracting patches, in order to cover the whole surface of the input image. In our experiments we found has almost the same accuracy. We argue that having large patch size makes our network invariant to small changes in . Note that small values of makes training time slower and the network prone to overfitting.

4 Experiments and Results

Our dataset is composed of high resolution Hematoxylin and Eosin (H&E) stained breast histology microscopy images labelled as normal, benign, in situ carcinoma, and invasive carcinoma ( images for each category). These images are patches extracted from whole-slide images and annotated by two medical experts. Images for which there was a disagreement between pathologists were discarded. Figure 4 highlights the variability in sample images. The dataset was available at

Figure 4: Samples of ICIAR 2018 challenge on BreAst Cancer Histology (BACH) dataset

The size of dataset is relatively small for our convolutional network. To prevent patch-wise network from overfitting, we apply several data augmentations on patches extracted from the microscopy image. Pathology images do not have canonical orientation and the classification problem is rotation invariant [12, 3]. To augment the dataset we rotate each patch by multiples of , with and without mirroring, which results in valid variations for each patch. We further apply random color perturbations to these variations as suggested by [20] and produce more patches. The color augmentation process would help our model to learn color-invariant features and make pre-processing color normalization [21] step unnecessary. The total size of our patch-wise dataset becomes . We train both networks on a single NVIDIA Titan XP GPU using Adam [22] optimizer, mini-batch size of and initial learning rate of , with a decay of every epochs. We train the patch-wise model on of our dataset for epochs and used the remaining for cross-validation. We use the same train/validation sets for image-wise network. The accuracy of the proposed method is measured as the ratio between correct samples and the total number of evaluated images. For our validation set of images, our best model achieved accuracy (Table 1) and mean Area Under Curve (AUC) of corresponding to (, , , ) for the four classes (Table 2) based on Receiver Operating Characteristic (ROC) analysis. In addition, we experimented with an ensemble model, averaging across variations of rotation/flip in the input image as suggested by [20] and further improved the accuracy to . To compare our results with those of Araújo et al.[3], the image-wise labels through a decision making scheme on outputs of the patch-wise network is also presented in (Table 1). The image label is obtained using one of the three different patch probability fusion methods. These methods include, majority voting, where the image label is selected as the most common patch label, maximum probability, where the patch with higher class probability decides the image label, and sum of probabilities, where the patch class probabilities are summed up and the class with the largest value is assigned. As shown in (Table 1), our proposed patch-wise network is outperforming previous methods by a large margin.

Patch-wise Image-wise
Sum Max. Maj.  CNN Ensemble
Proposed 91.25 92.50 90.00  93.75 95.00
Araújo et al.[3] 77.8 72.2 77.8 - -
Table 1: Accuracy of the proposed method for one-channel output compared to [3]

To further experiment the effect of using spatial average layer at the end of the patch-wise network, we use the hyper-parameter that controls the depth of the output feature maps. We examined different values of and measured the accuracy using our validation set (Table 2). In our tests, the network with only one output channel outperforms the others corroborating the idea that having many filters for a single concept impose extra burden on the next network, which needs to adjust with all variations from the previous network [17]. The ROC curves for and are illustrated in Figure 5.

Figure 5: Receiver Operating Characteristic (ROC) curves for two output sizes of the patch-wise network. Left: 1-Channel feature map, Right: 4-Channel feature map
Patch-wise Image-wise
Feature maps Class Precision Recall Accuracy Class Precision Recall AUC Accuracy
Normal 0.80 0.81 0.82 Normal 0.95 1.00 0.99 0.94
Benign 0.81 0.77 Benign 0.95 0.95 0.98
InSitu 0.76 0.85 InSitu 0.86 0.95 0.99
Invasive 0.93 0.86 Invasive 1.00 0.85 0.98
Normal 0.81 0.86 0.86 Normal 0.94 0.85 0.97 0.90
Benign 0.85 0.79 Benign 0.86 0.90 0.96
InSitu 0.86 0.88 InSitu 0.83 0.95 0.98
Invasive 0.93 0.90 Invasive 1.00 0.90 0.98
Normal 0.81 0.90 0.86 Normal 0.79 0.95 0.98 0.88
Benign 0.85 0.79 Benign 1.00 0.70 0.97
InSitu 0.90 0.84 InSitu 0.86 0.90 0.99
Invasive 0.90 0.93 Invasive 0.90 0.95 0.97
Normal 0.77 0.86 0.82 Normal 0.78 0.90 0.98 0.85
Benign 0.85 0.68 Benign 0.83 0.75 0.96
InSitu 0.80 0.87 InSitu 0.94 0.80 0.96
Invasive 0.86 0.85 Invasive 0.86 0.95 0.99
Table 2: Detailed results of both networks over different classes

5 Conclusions and Future Work

In this manuscript, we considered the problem of breast cancer classification using microscopy tissue images. We utilized deep learning techniques and proposed a novel two-stage CNN pipeline to overcome the hardware limitations imposed by processing of very large images. The first so called patch-wise network acts on the smaller patches of the whole image and outputs spatially smaller feature maps. The second network is performing on top of the patch-wise network. It receives stack of feature maps from the patch-wise network as input and generates image-level label scores. In this framework, patch-wise network is responsible for capturing the local features of the input while the image-wise network is learning to combine those features and find the relationship between neighbouring patches to globally infer characteristics of the image and generate class confident scores. The main contribution of this work is presenting a pipeline which is able to process large scale images using minimal hardware. We trained the networks using the ICIAR 2018 grand challenge on BreAst Cancer Histology (BACH) dataset. The proposed method yields accuracy on the four-class validation set compared to previously reported accuracy rates in [3]. It is worth noting that inference time of our scheme is in the order of milliseconds. The pre-trained weights of our networks are relatively small in size (MB patch-wise,

MB image-wise) compared to other state-of-the-art networks (hundreds of MB) which makes them suitable for practical settings. We trained two networks separately on the same labels with the same loss function. One might fairly argue that training the patch-wise network with the same labels as the image-wise network is a disadvantage to the performance of our model. Clearly, not every patch in an image represents the same category. When patches are extracted from a large image, there might be patches that their labels are different than the whole-image. For example, in a Carcinoma image, there might be patches that are ”Normal”, but we classify them using ”Carcinoma” because the labels for patches are not given! One alternative would be to train both networks end-to-end using only one loss function that back-propagates through both networks. In this scheme, both networks are interconnected to let the flow of gradient and therefore cost is minimized by updating both networks’ parameters together. In our experiments, we found that such model requires a very large memory footprint that makes it impractical to apply in practice. We plan to further investigate this framework in the future and focus on its improvements.


This research was supported in part by a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant (DG) for ME. AA would like to acknowledge UOIT for a doctoral graduate international tuition scholarship (GITS). The authors gratefully acknowledge the support of NVIDIA Corporation for their donation of Titan XP GPU used in this research through its Academic Grant Program.


  • [1] Siegel, R.L., Miller, K.D., Jemal, A.: Cancer statistics, 2016. CA: a cancer journal for clinicians 66(1) (2016) 7–30
  • [2] Ghaznavi, F., Evans, A., Madabhushi, A., Feldman, M.: Digital imaging in pathology: whole-slide imaging and beyond. Annual Review of Pathology: Mechanisms of Disease 8 (2013) 331–359
  • [3] Araújo, T., Aresta, G., Castro, E., Rouco, J., Aguiar, P., Eloy, C., Polónia, A., Campilho, A.: Classification of breast cancer histology images using convolutional neural networks. PloS one 12(6) (2017) e0177544
  • [4] Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H.: Patch-based convolutional neural network for whole slide tissue image classification.

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 2424–2433

  • [5] Cruz-Roa, A., Basavanhally, A., González, F., Gilmore, H., Feldman, M., Ganesan, S., Shih, N., Tomaszewski, J., Madabhushi, A.: Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. In: SPIE medical imaging. Volume 9041., International Society for Optics and Photonics (2014) 904103–904103
  • [6] 15th International Conference on Image Analysis, Recognition: ICIAR 2018 grand challenge
  • [7] Kowal, M., Filipczuk, P., Obuchowicz, A., Korbicz, J., Monczak, R.: Computer-aided diagnosis of breast cancer based on fine needle biopsy microscopic images. Computers in biology and medicine 43(10) (2013) 1563–1572
  • [8] George, Y.M., Zayed, H.H., Roushdy, M.I., Elbagoury, B.M.: Remote computer-aided breast cancer detection and diagnosis system based on cytological images. IEEE Systems Journal 8(3) (2014) 949–964
  • [9] Filipczuk, P., Fevens, T., Krzyzak, A., Monczak, R.: Computer-aided breast cancer diagnosis based on the analysis of cytological images of fine needle biopsies. IEEE Transactions on Medical Imaging 32(12) (2013) 2169–2178
  • [10] Brook, A., El-Yaniv, R., Isler, E., Kimmel, R., Meir, R., Peleg, D.: Breast cancer diagnosis from biopsy images using generic features and svms. IEEE Transactions on Information Technology in Biomedicine (2006)
  • [11] Zhang, B.: Breast cancer diagnosis from biopsy images by serial fusion of random subspace ensembles. In: Biomedical Engineering and Informatics (BMEI), 2011 4th International Conference on. Volume 1., IEEE (2011) 180–186
  • [12] Cireşan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Mitosis detection in breast cancer histology images with deep neural networks. In: International Conference on Medical Image Computing and Computer-assisted Intervention, Springer (2013) 411–418
  • [13] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  • [14] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift.

    In: International Conference on Machine Learning. (2015) 448–456

  • [15] Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. ICML. Volume 30. (2013)
  • [16] Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
  • [17] Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint:1312.4400 (2013)
  • [18] Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research 15(1) (2014) 1929–1958
  • [19] Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks. In: Advances in Neural Information Processing Systems. (2016) 4898–4906
  • [20] Liu, Y., Gadepalli, K., Norouzi, M., Dahl, G.E., Kohlberger, T., Boyko, A., Venugopalan, S., Timofeev, A., Nelson, P.Q., Corrado, G.S., et al.: Detecting cancer metastases on gigapixel pathology images. arXiv preprint arXiv:1703.02442 (2017)
  • [21] Macenko, M., Niethammer, M., Marron, J., Borland, D., Woosley, J.T., Guan, X., Schmitt, C., Thomas, N.E.: A method for normalizing histology slides for quantitative analysis. In: Biomedical Imaging: From Nano to Macro, 2009. ISBI’09. IEEE International Symposium on, IEEE (2009) 1107–1110
  • [22] Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)