Learning New Tricks from Old Dogs – Inter-Species, Inter-Tissue Domain Adaptation for Mitotic Figure Assessment

11/25/2019 ∙ by Marc Aubreville, et al. ∙ 23

For histopathological tumor assessment, the count of mitotic figures per area is an important part of prognostication. Algorithmic approaches - such as for mitotic figure identification - have significantly improved in recent times, potentially allowing for computer-augmented or fully automatic screening systems in the future. This trend is further supported by whole slide scanning microscopes becoming available in many pathology labs and could soon become a standard imaging tool. For an application in broader fields of such algorithms, the availability of mitotic figure data sets of sufficient size for the respective tissue type and species is an important precondition, that is, however, rarely met. While algorithmic performance climbed steadily for e.g. human mammary carcinoma, thanks to several challenges held in the field, for most tumor types, data sets are not available. In this work, we assess domain transfer of mitotic figure recognition using domain adversarial training on four data sets, two from dogs and two from humans. We were able to show that domain adversarial training considerably improves accuracy when applying mitotic figure classification learned from the canine on the human data sets (up to +12.8 method to transfer knowledge from existing data sets to new tissue types and species.



There are no comments yet.


page 2

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Mitotic figures, i.e. cells undergoing cell devision, are an important marker for tumor proliferation and their density is used in many tumor grading schemes for prognostication [1]. Fostered by the availability of datasets, this field has seen significant algorithmic advances in the very recent past. Especially in the field of human mammary carcinoma, one of the most common tumors in women, data sets like MITOS [2]

were able to include a great deal of variance that are typical in pathology. Since mitosis is a multi-phase process, the visual variability of its appearance in microscopy images is high. Besides this, there is a number of other factors that increase variance: staining is a manual or semi-automated process, and such is the preparation of the tissue sections to put on the microscopy slide. Another source of variance is the tissue itself, as especially macroscopic structures are differing considerably between tissue types. On top of that, we can expect differences between humans and other species - however, it could be well debated if, given a significant sources of variability, these would always play a major role.

All these factors cause a domain shift between data sets that is well-known in digital histopathology, and thus machine learning algorithms that have been trained on one data set will rarely directly transfer to another. Two classes of algorithmic adaptations to models are typically used in this regard: First, we can train a model on a large data set in one domain, which will allow to provide the model with a good initialization of its feature space for the next task, which is training the model with a (smaller) set of images in the target domain - typically referred to as

transfer learning. This, however, requires annotated images in the target domain, which are also not always available to a sufficient amount.

Another approach is domain adaptation, which can also be done unsupervised, i.e. without any annotation and just images available in the target domain. These approaches rely solely on statistical properties of the input distributions, and will only work sufficiently if the derived real labels of the respective known source domain and the (assumed to be unknown) target domain follow the same ruleset. This is especially tricky in the field of mitotic figure detections, where individual thresholds play a significant role and thus the labels are significantly dependent on the expert defining the label.

Recently, Li et al.

have shown, that a combination of an object detection network and a refining second stage classifier

[3] show superior performance in mitosis detection, which was in line with our own findings [4]. Due to this, it seems interesting to have a first stage identifying mitotic figures, which acts as a coarse classifier and do the fine classification in a secondary stage, where domain adaption is most crucial. Thus, this work focuses on the distinguishing of mitotic figures from similar looking cells (hard negatives), which would be the task of the second stage.

2 Material and Methods

Figure 1:

T-SNE representation of the various domains and classes in our data set, features extracted from a ResNet-18 stem, trained on ImageNet. Human data sets are from mammary carcinoma (MITOS2014) and meningioma (HUMMEN), canine data sets are from cutaneous mast cell tumor (CCMCT) and mammary carcinoma (CMC).

In this work, we use four data sets, two from canine histopathology slides and two from human histopathology slides. Our research group recently published a data set of canine cutaneous mast cell tumor (CCMCT, a common tumor in dogs), spanning 32 whole slide images and providing annotation for all mitotic figures on the complete slides [4]

. For this work, we have generated additionally a data set of canine mammary carcinoma (CMC) which includes 22 tumors also completely annotated blindly by two expert pathologists and with consensus found for all disagreed cells using an open source solution

[5]. As for human tissue, we set up a pilot data set of human meningioma (HUMMEN) consisting of three whole slide images from WHO grades I, II and III. All tissue was sampled for routine diagnostics and, where applicable, institutional review board approval and written consent was given (IRB approval number hidden for review). Additionally, to set a baseline, we used the publicly available MITOS2014 data set [2], which contains mitotic figure annotations on human mammary tissue. All four data sets have in common that they contain not only true mitotic figure annotations, but also annotations that can easily be misinterpreted by either an algorithm or a human, but were in final consensus of all experts not found to be mitotic figures. These can be considered truly hard negative, and thus the differentiation between this group and the group of mitotic figures is one of the hardest tasks in mitotic figure detection. As Fig. 1 shows, the domain shift between the data sets can be nicely be spotted in a t-SNE representation, while the cell label is not easily distinguishable.

data set name species tumor cases mitotic figures mitotic figure look-alikes annotations available
CCMCT [4] dog cutaneous mast cell tumor 32 44,880 27,965 complete WSI
CMC (unpublished) dog mammary carcinoma 22 13,385 26,585 complete WSI
MITOS 2014 [2] human mammary carcinoma 11 749 2,884 selected area
HUMMEN (unpublished) human meningioma 3 782 1,721 complete WSI
Table 1: Comparison of the data sets used in this study.

Domain-adversarial training [6] is an unsupervised network adaptation method, recently also employed in the field of histopathology [7]. It aims to account for a domain shift in latent space that is known to occur between slides of the same data set, but also, and more severely, between data sets. The core idea is to train a model with a body and two heads, where one head aims to perform classification of the cell class, and the other would aim for the classification of the domain. The latter inherently uses said domain shift to differentiate between data sets, however, it is this domain shift that we want to reduce.

Ganin et al. proposed to use gradient reversal during network training for this task [6]. The gradient reversal layer introduced by them acts as a pure forwarding layer during inference, but inverts the sign of all gradients during back-propagation. We aim to use the method to make the latent space representations for cell classification indistinguishable between source and target domain, and thus improve classification for the target domain in an unsupervised way. Using squared input images of 128px size with the respective cell at it’s center, we employed a state-of-the-art network stem (ResNet-18 [8]

) for the main feature extraction. After flattening the feature vector, we have two linear (fully connected) layers each with batch-norm and ReLU activation after the first layer as the cell label head (Fig.


). The secondary domain classification head starts with a gradient reversal layer (GRL) and another fully connected layer with batch normalization. Afterwards, we concatenate the intermediate features of the classification head into the secondary branch, to also be able to reduce domain shift in this layer (as proposed by Kamnitsas et al. 


Figure 2: Domain adversarial training of our network.


For training, we feed all images, and suppress the cell label information (mitosis/nonmitosis) in the loss function for samples of the target domain.We did not use a validation set for model selection, as a train/val/test split on patient level for that purpose is questionable as the domain shift will likely be also high within the data set. We chose an even distribution of cell labels and domains for both, the training and the test set.

We denote the output probability of the cell classification branch and the domain classification branch

and , respectively. The true cell class and domain are denoted as and , respectively. We derive the domain classification loss as simple cross-entropy loss:


With representing the target domain, the cell classification loss is:


The total loss is a normalized weighted sum of both, i.e with uppercase symbols denoting the respective mini batch vectors



is a tunable parameter. We trained for 3x30 epochs using super-convergence 

[10] and Adam as optimizer. For better comparability, we limited the number of cells for the large data sets to 1,600.

3 Results

(a) Box-whisker plots of cell label accuracy
(b) Feature vector (t-SNE)
Figure 3: Results of domain transfer. Plot a) shows accuracy for 5 runs on the different domain transfer tasks, whereas b) shows an example ResNet-18-feature vector with and without domain transfer (CCMCT to MITOS2014 task).

Domain transfer raised the mean accuracy over 5 runs to on the HUMMEN data set by 4.7 percentage points, on the MITOS2014 data set by 12.8 percentage points and in total by 6.5 percentage points (Fig. 3(a)). This improvement can largely be attributed to a reduced domain shift in latent space (Fig. 3(b)). The performance varied both for the domain-adapted and the non-adapted case, and was especially large, where the original domain shift was most severe (MITOS2014 target data set). In this case, we achieved the best results when training on the same tissue type for a different species (CMC).

4 Discussion

Our results indicate that domain adversarial training is a suitable unsupervised learning method to perform mitotic figure domain adaptation between tissue and, equally importantly, also species. This is especially beneficial as publicly available, large datasets can be used for new domains - regardless of tissue/tumor type or species (animal or human). Further, we find a clear dependency on source and target data set, which is not surprising considering the initial latent space distributions.

While the results showed clear benefits for the given task, we should point out our method needs access to pre-selected mitotic figures candidates, which we extracted beforehand. This step could, however, be taken over by generalizing model that was trained on a broad range of tissue, which we aim to investigate in future research.


CAB gratefully acknowledges financial support received from the Dres. Jutta & Georg Bruns-Stiftung für innovative Veterinärmedizin.


  • [1] C W Elston and I O Ellis. pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology, 19(5):403–410, November 1991.
  • [2] Roux, L, Racoceanu, D, Capron, F, Calvo, J, Attieh, E, Le Naour, Gilles, and Gloaguen, Anne. MITOS & ATYPIA - Detection of mitosis and evaluation of nuclear atypia score in breast cancer histological images . IPAL, Agency Sci., Technol. Res. Inst. Infocom Res., Singapore, Tech. Rep, 2014.
  • [3] Chao Li, Xinggang Wang, Wenyu Liu, and Longin Jan Latecki. DeepMitosis: Mitosis detection via deep detection, verification and segmentation networks. Med Image Anal, 45:121–133, April 2018.
  • [4] Christof A Bertram, Marc Aubreville, Christian Marzahl, Andreas Maier, and Robert Klopfleisch. A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci Data, 274:1–9, 2019.
  • [5] Marc Aubreville, Christof Bertram, Robert Klopfleisch, and Andreas Maier. Sliderunner. Procs BVM, pages 309–314, 2018.
  • [6] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S Lempitsky.

    Domain-Adversarial Training of Neural Networks.

    J Mach Learn Res, 2016.
  • [7] Maxime W Lafarge, Josien PW Pluim, Koen AJ Eppenhof, Pim Moeskops, and Mitko Veta. Domain-adversarial neural networks to address the appearance variability of histopathology images. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 83–91. Springer, 2017.
  • [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Procs CVPR, pages 770–778, 2016.
  • [9] K Kamnitsas, C Baumgartner, Christian Ledig, Virginia F J Newcombe, Joanna P Simpson, Andrew D Kane, David K Menon, Aditya V Nori, Antonio Criminisi, Daniel Rueckert, and Ben Glocker. Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. Int Conf on Inf Proc in Med Imaging, pages 597–609, 2017.
  • [10] Leslie N Smith and Nicholay Topin. Super-convergence: very fast training of neural networks using large learning rates. In Tien Pham, editor, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, page 1100612. International Society for Optics and Photonics, May 2019.