Cervical cancer is one of the most common causes of cancer death for women worldwide and is most frequently in developing countries william2018review . Papanicolaou test (abbreviated as Pap test) or cervical cytology is now a mainstay cervical cancer screening method to detect potentially pre-cancerous and cancerous process in the cervix, which has demonstrated reduction in cervical cancer incidence and mortality in developed countriesmishra2011overview ; bora2017automated . Such method is performed by a visual examination of cytopathological analysis under the microscope of the glass slide and finally giving a diagnosis report according to the descriptive diagnosis method of the Bethesda system (TBS) nayar2015bethesda . However, manual analysis of microscope images is time-consuming, labor-intensive and error-prone as a handful of abnormal cells among millions of cells within a single slide has to be identified by a trained professional william2019pap .
Therefore, automation-assisted screening based on cervical cytology has become a necessity. Since the first system was developed in 1950s, extensive research has attempted to exploit automation-assisted reading systems based on automatic image analysis techniques(e.g.birdsong1996automated ; koss1994evaluation ; wilbur1998autopap ) which led to a couple of commercial systems emerged, such as the BD FocalPoint Slide Profiler wilbur2009becton and ThinPrep biscotti2005assisted which received approval from the US Food and Drug Administration(FDA). While automation-assisted reading systems can increase productivity by reducing the time needed to read slides, their current performance and costs are not recommended for application in primary cervical screening kitchener2011automation ; bengtsson2014screening . To this end, lots of automation-assisted methods based on cervical cell image analysis have been proposed william2019pap ; gencctav2012unsupervised ; zhang2014automation ; sarwar2015hybrid ; sharma2016various . Most of them follow the multi-stage pipeline, i.e., first identifying the candidate regions based on segmentation, then extracting hand-crafted features based on the characteristics of nuclei and cytoplasm for classification, as shown in Fig. 1 (a).
While most of these studies have achieved available performance whether in cell segmentation or cell classification, there still have some challenges to use them in clinical automation-assisted reading. First, the current automation-assisted reading approaches has not been sufficiently cost-effective to promote to the cervical cell screening in clinical due to the tedious image patches preprocessing and screening process for cyto-technicians and doctors. The majority of existing research has done on the Herlve dataset jantzen2005pap which only contains single-cell images with a size of 200100 pixels approximately and was produced carefully by trained professionals. As shown in Fig. 2, images from Herlve dataset are all clear with no overlapping and impurity. In fact, the slide image with around 2,160 million pixels obtained by the Whole Slide Imaging (WSI) technology Fertig2017Whole has complex cellular situation, such as cell overlapping, noise and impurity. Thus,one original cervical cell slide should be cropped into a huge number of single-cell image patches by using sliding windows or region proposal generation methods based on low-level image features, which lead to low efficiency. Second, it is difficult to make segmentation of the cytoplasm and nuclei absolutely due to the high degree of cell overlapping, the poor contrast of the cell cytoplasm and the presence of mucus, noise and impurity. Third, it is worth considering that whether the hand-crafted features can represent complex identification information or not since the richer semantic information sensitive to recognition may actually exist in hidden upper-level features of cervical cell imageszhang2017deeppap . In addition, medical images are too complex and variable to get a perfect annotation as ground truth, which leads to noisy label unavoidably. However, the previous studies usually deal with cervical cell segmentation and classification without considering the existence of noisy labels.
To cope with these problems, we propose to utilize CNN-based object detection to automatically extract and learn task-specific features, and achieve the cervical cells recognition efficiently on multi-cell images with cell overlapping and clusters, as showing in Fig. 1 (b). Our method directly operates on multi-cell image with size of 40003000 automatically extract more complex discriminative features and can obtain a image-level classification results. Moreover, it not only achieves cervical cell image-level classification but also provides more detailed location and category reference information of abnormal cells. In detail, we exploit YOLOv3 redmon2018yolov3 as our cervical cell object detection baseline model due to the efficiency, accuracy and flexibility. In order to improve the classification performance of hard examples which are four highly similar categories, we cascade a further task-specifical classifier. Furthermore, we weaken the influence of noisy labels by smoothing their distribution.
Our contributions are summarized as follows, 1) Unlike the previous method, we treat the cervical cell recognition as object detection which automatically detect cervical cells directly on multi-cell images. It is more efficient as we can extract features automatically without manual intervention and careful design for all stages. Our method not only achieves cervical cell image-level classification but also provides more detailed location and category information of abnormal cells simultaneously. 2) We propose a simple and effective scheme, cascade a further task-specific classifier to improve the performance of hard example recognition. 3) We investigate the existence of noisy labels on cervical cell dataset and propose an approach to weaken the influence of them by smoothing their distribution.
2 Related Work
2.1 Cervical Cell Recognition
From the analysis of the existing work, extensive research chankong2014automatic ; garcia2016multiscale ; sun2017cervical ; kudva2018automation ; asiedu2018development has been devoted to the field of automatic recognition of cervical cancer and have obtained good results. The previous cervical cell recognition can be classified into two types according to the number of cells in the image: recognition based on single-cell image and multi-cell image.
Early methods proposed to achieve the automatic segmentation and classification of abnormal cervical cells based on isolated cell without overlapping images. The most of them used one or multiple techniques including thresholding gencctav2012unsupervised , morphology operation bamford1996water ; anantha2017segmentation
, k-meanstsai2008nucleus , Hough transform bergmeir2012segmentation and watershed plissiti2010automated . For the better ways, Li et al. li2012cytoplasm
utilize a Radiating Gradient Vector Flow (RGVF) Snake to extract both the nucleus and sytoplasm from a single-cell cervical cell image. After preprocessing, the areas in the image are roughly clustered into nucleus, cytoplasm and the background by a spatial K-means clustering algorithm. And then making segmentation of the image by using RGVF after extracting the initial contours. In addition, this work was further improved inguan2014accurate , they proposed a dynamic sparse contour searching algorithm to locate the weak contour points of cytoplasm in overlapping regions, and the Gradient Vector Flow Snake model is finally employed to extract the accurate cell contour based on the located contour points. In order to eliminate the limitations of hand-crafted feature, zhang2017deeppap ; yarlagadda2019system ; wieslander2017deep
used deep learning for cervical cell classification on single-cell images by extracting and learning features automatically, while the image preprocessing of them rely on effective techniques to deal with the recognition of the nucleus. However, The majority of existing research has done on the Herlve dataset which only contains single-cell images with a size of 200100 pixels approximately and was produced carefully by trained professionals. Images from Herlve dataset are all clear with no overlapping and impurity, which is different from gigapixel pathological slide in clinical that contains thousands of cervical cells.
Different from recognition based on single-cell image, it is more sophisticated to segment nuclei and cytoplasm of images containing a large number of isolated cells and cell clusters, especially overlapping cells, and this has attracted increasing research interestssong2016accurate ; Lu2017Evaluation ; lu2015improved ; lu2013automated ; arya2018clustering . William et al. william2019cervical trained a pixel level classifier on cell nuclei, cytoplasm, background and debris using a Trainable Weka Segmentation(TWS) toolkit arganda2017trainable to identify and segment different objects on a slide. Genctav et al. gencctav2012unsupervised proposed multi-scale hierarchical segmentation algorithm to partition the image into regions, and used binary classifier to segment cell regions into nucleus and cytoplasm. For better nucleus segmentation, Zhang et al. zhang2014segmentation used graph cut approach to segment cervical cells in images with healthy and abnormal cells, and then segmented the nuclei especially abnormal nuclei by using group cut adaptively and locally. In wang2019automatic , first obtaining regions of interest for cell nuclei segmentation by applying the Mean-Shift clustering algorithm, and then applying mathematical morphology to split overlapped cell nuclei for better accuracy and robustness.
Although these methods work well for the recognition of cervical cells, it is complicated that all stages should be designed carefully. Furthermore, accurate segmentation of cytoplasm and nucleus for the cervical cell is still particularly challenging due to the complexities of cell structures and image characteristics. Therefore, the recognition of cervical cell based on automatically learning rather than hand-designed is necessary.
2.2 CNN-based Object Detection
In the last years, deep learning is emerging as the leading automatic learning tool in the imaging and computer vision domains generally. Particularly, the CNN-based object detection has the widespread application value and the prospects for development in the field of computer vision tasks. The Overfeatsermanet2013overfeat
is a pioneer which applies CNN to do object detection and has won the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013(ILSVRC2013)russakovsky2015imagenet . Plenty of methods based on CNN have been proposed for object detection consecutively and have achieved significant advances in the last years. The current CNN-based detectors of state-of-the-art can be divided into two categories: (1) the two-stage approach, and (2) the one-stage approach.
In the two-stage approach, a sparse set of candidate object proposal is generated first, and then classify and regress them using convolutional networks. In particular, the famous two-stage approach(e.g., R-CNN girshick2014rich , SPPnet he2015spatial , Fast R-CNN girshick2015fast and Faster R-CNN ren2015faster ) has been achieving outstanding performances on several challenging benchmarks. Different from the two-stage approach which performs well in accuracy, the one-stage approach detects objects by regular and dense sampling over locations, scale and aspect ratios. The recently one-stage detectors, such as YOLOv3 redmon2018yolov3 , SSD liu2016ssd and the improvements jeong2017enhancement ; fu2017dssd ,are devoted to high efficiency but most of their accuracy are still yielding to the two-stage detectors.
Recent results have proven that the generic descriptors extracted from CNNs are extremely effective in object recognition and localization in natural images. Extensive research (e.g. dou2016automatic ; teramoto2016automated ; liang2018object ; bharath2018multi ; liang2018end ) has been devoted to the recognition of medical image domain based on deep learning methodologies soon. These methods automatically detect potentially lesion and have been shown to produce impressive and encouraging results. Specifically, CNN-based methodologies have also been used in cervical cell recognitionzhang2017deeppap ; wu2018automatic ; yarlagadda2019system ; wieslander2017deep , and have achieved good results. However, most of these approaches only used CNNs to classify images with single cell by feeding the network cropped patches they need. In this paper, we treat the cervical cell recognition as object detection and exploit YOLOv3 redmon2018yolov3 as our cervical cell object detection baseline model, to detect cervical cells directly on multi-cell images and then label the suspicious cells containing location, category, and corresponding confidence score.
The pipeline of proposed method includes cervical cell detection and hard example classifier, as show in Fig. 3. In this section, we firstly describe the pipeline of proposed end-to-end CNN-based object detection. And then, we introduce several improvements such as hard example classification, smoothing noisy label regularization, to make it more appropriate for our cervical cell recognition.
3.1 YOLOv3-based Cervical Cell Detector
Most state-of-the-art CNN-based detectors perform well, we accordingly exploit YOLOv3 redmon2018yolov3 with the input size of 416416 as our cervical cell object detection framework due to the efficiency, accuracy and flexibility. YOLOv3 redmon2018yolov3 can be decomposed into two parts, a custom deep architecture Darknet-53, which has 53 layers network trained on ImageNet deng2009imagenet , to extract features, and multi-scale feature fusion layers used as feature maps and predictors. In detail, first obtaining feature maps from input image with feature extractor, which is consisted of convolutional layers and residual blocks. Next, predicting on the Yolo layers at three scales, which are given by down-sampling the dimensions of the input image by 32,16 and 8
separately. In fact, small objects are the widespread in our dataset such as vaginalis (abbreviated as VAG), and are difficult to detect well. In order to detect small objects better, the three-scale feature maps fused by up-sampling layers and concatenate layers help preserve the fine-grained features. Finally, we get a 3-d tensor encoding bounding box, objectness score, and class predictions.
3.2 Cascade Hard Example Classifier
Cancerization of cells is a continuous process, and many abnormal cells are visually similar to each other. According to section 4.1, the dataset used in this paper contains 10 categories objects from cervical cell images, and the differences between these categories are small, which increasing the difficulty for network to extract more discriminative features and distinguish different cells morphology objects. Especially the four squamous cells categories: atypical squamous cells-undetermined significance (ASC-US), atypical squamous cells-cannot exclude HSIL (ASC-H), low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), which are still hard to be distinguished well after object detection networks. As showing in Fig. 4, differences between these hard example cells are small, and the scales of these cells are diverse after the image pyramid strategy (more details according to section 4.1.1). From the perspective of feature extraction, we can redesign a more fitting backbone network of our dataset to replace Darknet-53. However, it is difficult to obtain a pre-trained model with task-specific identification capability due to the limited dataset and hardware resources. After analysis, cascade a task-specific classifier can also replace the complex backbone network and achieves good performance.
To this end, we implement an object detection and classification cascade framework aiming to improve the hard example identification performance. To be specific, we use InceptionV3 base model szegedy2016rethinking , with weights pre-trained on ImageNet deng2009imagenet
, to achieve good balance between speed and accuracy. On top of the InceptionV3, we attach a Global Average Pooling layer, a fully-connected layer with 1024-dimensional output channels followed by the ReLU function and a fully-connected layer with 4 channels (the number of subclass) followed by the softmax function. In this way, we can obtain more accurate classification of these four classes after the object detection network. As illustrated in the right of Fig.3, during testing, for one image which outputted from the cervical cell detector, we just select all bounding boxes which are predicted as the four hard example categories and feed them into cascade classifier to achieve higher accuracy, then feedback the classification results to detection results.
3.3 Smoothing Noisy Label Regularization
Unlike natural images, the medical images are too complex and variable to obtain a perfect annotation as the standard ground truth. In fact, cancerization of cells is a continuous process, there is no absolute threshold for quantitative discrimination that which stage the current cell belongs to. From section 3.2 we know that most cells in our dataset are too similar to differentiate them well in most cases whether in clinical or in computer-assisted recognition system at present, especially experts may have different opinions. Therefore, it is extremely tedious and highly subjective to mark hand-crafted labels for cells, which leading to noisy label inevitably.
To cope with this problems, we modify the distribution of our noisy labels to regularize the classifier layer instead of pursuing perfect manual annotation. During training, the input image is computed by the CNNs and gets the confidence score of the current input image corresponding to each category. These scores are normalized by softmax function, and the probability that the current input image belongs to each category is finally obtained:
Here, are the or unnormalized log-probabilities and is the total number of classes. We set to 10. And then utilize cross-entropy function to calculate loss:
is the ground-truth one-hot distribution of label
, where the truth class has probability one while all other classes have zero. Finally, minimizing the cross-entropy of prediction probability and the true probability of label, so as to obtain the optimal prediction probability distribution during training the network. Only whencan softmax function approach this distribution, however, it never reach it. This encourages the model to be too confident about its predictions and leads to over-fitting. Due to the unreliable manual annotation, our model will learn in the wrong direction, for example, the true category of one object is ASC-H, while the manual annotation of it is ASC-US, then our model believe that features extracted from it are belong to ASC-US, but actually not. Label smoothing was proposed in szegedy2016rethinking as a form of regularization. Specifically, for a single ground-truth label , we change the probability of true class into while all other classes becomes . We smooth the ground-truth label distribution with
where is a small constant. This method can partially reduce the confidence of our model and weakens the influence caused by our noisy labels through suppressing the output difference between positive and negative examples.
4 Experiments and Discussion
4.1 Image Dataset
As there is no standard clinical cervical cells dataset with multi cells available publicly, we establish our own dataset captured by digital camera Ximea MC124CG-SY-UB with 12 million pixels situated on the microscope Olympus BX40 with 20 objective. Each pixel has a size of 3.45 . For one cervical cell slide, we can capture about 1,800 images. The specimens were prepared by liquid-based cytology with Feulgen staining. The dataset used in this paper is consisted of 12,909 cervical images with 58,995 ground truth boxes and contains 10 categories objects from cervical cell images, i.e., normal cells (NORMAL), atypical squamous cells-undetermined significance (ASC-US), atypical squamous cells-cannot exclude HSIL (ASC-H), low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), atypical glandular cells (AGC), adenocarcinoma (ADE), vaginalis trichomoniasis (VAG), monilia (MON) and dysbacteriosis (DYS). The ground truth boxes are marked by four trained cyto-technicians and an experienced pathologist, in order to maximize certainty and accuracy of the diagnosis. Examples of 10 categories cervical cells and the visible difference between subcategories are shown in Fig. 5, we can find that highly similar visual features of inter-class and intra-class are widely exist, such as ASC-H, ASC-US, LSIL and HSIL.
4.1.1 Acquisition of Image Patch
As the images of our dataset are captured with size of 41123008 on the WSI, it is not suitable for the large annotated images to input into our network of which the input size is 416416, since many deep identification information may lose after compressing and down-sampling in the network. Moreover, enough dataset is crucial to the high performance of CNN. However, as high expertise is required for quality annotation, there is a very limited amount of annotated data for cervical cells available. To deal with these problems, we propose the image pyramid strategy to split the microscopical images into several patches with equally size of 800600 instead of simple and crude resize operation. In detail, we first resized the original images into three scales,40003000 (layer one), 16001200 (layer two) and 800600 (layer three). Then we equally split layer one and layer two into 25 image patches and 4 image patches separately. The size of each patch is 800600 pixels. Therefore, we got a total of 30 image patches finally. Our image pyramid strategy can expand training samples and obtain multi-scale objects, thereby improving the performance of CNN.
As showing in Fig. 6, the scale of all cells varies widely from 600800 to 1010, which brings challenge to our work since smaller objects are difficult to extract richer semantic information and recognize well. After image pyramid strategy, we obtain total 66,627 image patches with size of 800600 and 138,314 ground truth boxes with multi-scale objects, including 21,388 for NORMAL, 19,879 for ASC-US, 13,616 for ASC-H, 9,092 for LSIL, 16,711 for HSIL, 20,874 for AGC, 2,930 for ADE, 18,173 for VAG, 9,622 for MON, and 6,029 for DYS. From the final datasets, we randomly select 1/10 as test set, and the others as trainval set, where train set makes up 4/5. Fig. 7 demonstrates the details of our dataset organization and categories distribution. The tricolor stacked histogram shows the object quantity of train/ val/ test sets, respectively.
4.2 Evaluation Metrics
As to cervical cell detection evaluation metrics, we follow the evaluation metrics used by the PASCAL VOC object detection challengeeveringham2010pascal , which are Average Precision (AP) and mean Average Precision (mAP). In this paper, we use AP at a single Intersection over Union (IoU) threshold of 0.5. Since cervical cell detection is a multiple classification task, we calculate AP for detection performance of our model separately for each category. In detail, for a given class, the precision/recall curve is calculated from the ranked ouput of our network. Recall is defined as the proportion of all positive instances ranked above a specific rank. Precision is the proportion of all instances above that rank which are from positive class. The AP summarises the shape of the precision/recall curve, and is defined as the mean precision at a set of eleven equally spaced recall levels[0, 0.1,…,1]everingham2010pascal . And mAP is the mean of all categories of AP. In addition, we use accuracy (Acc), the global percentage of correctly classified cells, to evaluate the performance of our hard example classifier. As to cervical cell image-level classification, we evaluate the performance of it on our test set with size of 40003000 using accuracy (Acc), sensitivity (Sens) and specificity (Spec), where Sens means the proportion of correctly classified abnormal images, Spec means the proportion of correctly classified normal images, and Acc is the total percentage of correctly classified images.
Note that the categories ASC-H and HSIL are too similar to distinguish them confidently in most cases whether in clinical or in computer-assisted recognition system at present, especially experts may have different opinions. To our knowledge, the result that mutual recognition of these two categories is acceptable to the clinical experts. One strategy to cope with the different opinions of experts is to invite some experts to simulate the balance point. We random select 60 cervical cell images which totally contain 52 cells (maybe ASC-H or HSIL), among them 20 objects were predicted as ASC-H (pred-ASCH cells), 34 objects were predicted as HSIL (pred-HSIL cells) by our methods. And then five experts vote the category that each cell belongs to. Finally, for the cells that were predicted as ASC-H, the experts vote 49 ballots to ASC-H and 51 ballots to HSIL, and for the rest cells, the experts vote 112 ballots to ASC-H and 58 ballots to HSIL. In other words, for one cell that was predicted as ASC-H (or HSIL) while the annotated label is HSIL (or ASC-H), we think that our prediction is not absolutely wrong. In order to obtain more reliable and appropriate result for cervical cell recognition, we modify the evaluation metrics by setting TP=0.51 instead of zero when the detection result is ASC-H but the truth label is HSIL, and correspondingly, setting TP=0.66 instead of zero when the detection result is HSIL while the correct label is ASC-H. Unless otherwise specified, all results are calculated according to the original evaluation metrics.
4.3 Cervical Cell Recognition Based on YOLOv3
We first apply Darknet-53, which originally has 53 layers network trained on Imagenet, as our convolutional feature extractor, to the input image to obtain high-level features. And then fine-tuning all convolutional layers of YOLOV3 on our cervical cell datasets. During training YOLOV3 model, we freeze the first 185 layers of total 252 layers to get a stable loss and the batch size is 32. We started from a learning rate 0.001, and total train 30 epochs. At the time of fine-tuning, we unfreeze all of the layers, with the batch size of 8, and adopt a learning rate policy of plateau: 0.0001 base learning rate, 0.1 factor and patience of 3 (means after 3 epochs while the performance of model does not improve, the learning rate reduction action will be triggered). Generally, we adopt data augmentation, such as rotation, resize, vertical flip and horizontal flip, in our dataset to expand training samples, avoid over-fitting and improve accuracy. The next series of YOLOv3 experiments are all adopt these strategies and test on 1 NVIDIA GTX1080Ti GPU.
In order to validate the effectiveness of our methods for cervical cell detection, we compared the detection performance of other networks in term of accuracy and speed. Results are reported in Table 1. For the sake of fairness, all these networks did not add any tricks. We find that YOLOv3 takes less test time per image and the mAP is better than the two-stage network FasterR-CNN. We think that the ability of FasterR-CNN to learn multi-scale objects is weak due to the lack of FPN lin2017feature structure. We also discussed the input scale and model size of network. Generally, increasing the size of input images can improve the accuracy, so we increase the input size from 416416 to 608
608. Unfortunately, we just obtain 57.4% mAP while cost 99 ms per image.. Tiny YOLOv3 is generated by removing the residual connectionhe2016deep and one yolo layer from YOLOV3. Although the speed is faster than others, the mAP is too worse. We argue that due to the reduction of Yolo layer and simplification of feature extractor, the ability of Tiny YOLOv3 to extract more fine-grained features on multi-scales is decreased.
It is typical to have a collection of boxes overlaid on the image at different spatial locations, scales and aspect ratios that act as ”anchors” (also called ”priors” or ”default boxes”) huang2017speed . Generally, YOLOv3 redmon2018yolov3 used k-means clustering on the COCO dataset lin2014microsoft to determine their bounding box priors as anchor boxes. Unlike generic objects in natural images, the objects of cervical cells vary very widely in their shapes, sizes and numbers, which lead to poor location and regression performance of potential instances. Therefore, we used k-means clustering to generate 9 anchors based on our dataset again (called ”re-anchor” by the authors). Our new 9 clusters were: (7 11), (12 19), (17 26), (26 36), (32 52), (47 64), (60 94), (92 127), (144 208). Table 2 shows that our new anchors are more adaptable and robust in our dataset than the original anchor that generated based on COCO, and yields higher mAP.
4.4 Ablation Studies
For all ablation studies, we use an image scale of 800600 for both training and testing. We evaluate the contribution of several important elements to our methods, including hard example classification, smoothing noisy label regularization, and re-anchor. The performance of our methods are all calculated on test specimens with size of 40003000. Results as reported in Table 2.
4.4.1 Hard Example Classification
, to achieve good balance between speed and accuracy. During training, we fine-tune all Inception modules by freezing the first 17 layers of total 314 layers after attaching our classification layers and the batch size is 32. And also adopting a learning rate policy of plateau: 0.0001 base learning rate, 0.1 factor and patience of 5 epochs. The input size of the network is 299299. As there are no original classification datasets of cervical cell, we produce the classification datasets by cropping the original annotated detection images according to their ground truth bounding box. In fact, our classification dataset is consisted of 16,332 squamous cell images as train set, 2,082 squamous cell images as validation set and 2447 as test set, which contain 4 categories, ASC-US, ASC-H, LSIL, HSIL. Commonly, we also adopt data augmentation, such as rotation, shift, shear, zoom, and random horizontal flip, in our train set.
As expected, simply cascade the object detection and hard example classification greatly improves the mAP by 1.7%, especially ASC-H and LSIL on which improve the AP by 5.9% and 14.5% respectively, shown in Table 2. The further task-specific classifier can extract and learn more discriminative features on these hard examples. What’s more, we find that the hard example classification after detection networks with our new anchor boxes can improve the mAP by 2.1% (the seventh row of Table 2). As our new anchor boxes are more adaptable and robust in our dataset, they are beneficial to the location accuracy and accelerate convergence. Similarly, we also make experiments about varies classification model aiming to select the optimal model that is appropriate to our dataset. We fine-tune Xception chollet2017xception , MobileNet howard2017mobilenets and VGG19 simonyan2014very as a base model to train our classification and adopted the same training strategy with InceptionV3. Table 3 presents the comparison, from which we can see that VGG19 performs poor than InceptionV3, the model size is 6bigger. Although the Acc of Xception chollet2017xception is better, the training time is double than InceptionV3 based on the similar model size as decreasing the batch size setting from 32 to 16 to run Xception chollet2017xception in limited GPU resources.
4.4.2 Smoothing Noisy Label Regularization
Instead of pursuing perfect manual annotation, we can modify the distribution of our noisy labels to regularize the classifier layer. We smooth the ground-truth label distribution with Eqn. 3 where is a small constant. In order to achieve higher incremental in our dataset, we make several experiments with different . The comparative results are listed in Table 4, which shows that yield higher mAP in general. As stated in Table 2, smoothing noisy label regularization performs well on our dataset. It improves the mAP by 1.2% and 0.7% on the baseline and our network cascade with hard example classifier respectively. Finally, with re-anchor, the mAP is further improve to 60.2%. We make the observation that improvement mainly comes from the more accurate classification on hard example and more accurate localization. We also find that smoothing noisy label regularization and re-anchor can work well together. Applying smoothing noisy label regularization with new anchor boxes can improve the mAP by 2.0%. We argue that clustering on our dataset makes the network selects more appropriate priors anchor boxes which can improve the location accuracy and accelerate convergence. And modify the distribution of our noisy labels and further cascade classifer can improve the classification performance. Note that the AP of ASC-H and HSIL improves by 22.8% and 9.8% respectively using our improved evaluation metrics (the last row in Table 2). We demonstrate that mutual recognition between them is ordinary.
4.5 Classification on Cervical Cell Images
In order to evaluate the image-level classification performance of our method on cervical cell dataset, we calculate the Acc, Sens, and Spec evaluation metrics on test set. According to the Bethesda system (TBS) for reporting cervical cytology nayar2015bethesda , precancerous lesions and cancerous lesions as positive samples in our test set include five types: ASC-US, LSIL, ASC-H, HSIL, AGC and ADE. In detail, our test set contains total 1,014 cervical cell images with size of 40003000, which are consisted of 728 abnormal cell images (positive samples) and 286 normal cell images (negative samples). We test both on our baseline model and totally improved method above. From Table 5 we can see that the accuracy and sensitivity are high both on our baseline model and improved method, which illustrate the feasibility of our idea on image-level classification. In detail, we achieve a Acc with 89.3% and a perfect Sens with 97.5%, which is higher than most of previous approaches. Moreover, different from most of the traditional approaches operated on single-cell images, the primary screening results given by us is calculated on multi-cell images with size of 40003000. In other words, we can provide image-level assisted reference information to cytotechnologists and doctors instead of cell-level, which boosts efficiency of cervical cell primary screening. We make the observation that the Spec value is low with 67.8%. We consider such poor results mainly from the extremely imbalanced data distribution (number of positive samples is 3 than negative samples), which induces the model to identified more cells as abnormal. However, in the practical application for automation-assisted primary screening in clinical, high Sens even with fairly low Spec is acceptable for image-level screening due to all positive samples will be reexamined by doctors or cyto-experts. Furthermore, our improved methods substantially decreases FP and increases Spec by 5.6%. Therefore, we believe that the performance of our method on image-level classification is satisfactory and inspiring.
In this paper, we utilize object detection method to achieve the automation-assisted cervical cell reading system. Different from the multi-stage traditional approaches, which rely on the accuracy of segmentation and the efficiency of hand-crafted features, our method extract high-level features automatically and detect cervical cells directly. We exploit YOLOv3 as a base model to detect 10 categories and then cascaded a further hard example classifier to refine the 4 categories: ASC-US, ASC-H, LSIL, HSIL. Finally, we also investigated the presence of the noisy label and deal with them by smoothing their distribution. After conducting comprehensive experiments, the image-level classification performance on cervical cell test set with size of 40003000 is excellent with 89.3% Acc,97.5% Sens and 67.8% Spec. Particularly, not only dose our method achieve the automatic cervical cell image-level classification screening but also it can detect the categories and location of abnormal cells. We achieve a mean average precision(mAP) of 60.2% in our dataset, and improve the AP of hard examples which are the most valuable but most difficult to distinguish. The results indicate that the performance of our automatic detection method provides a good reference and basis for the next work. We hope that our methods can provide a new idea for future development of computer-assisted reading systems for cervical cell primary screening and diagnosis in clinical.
Conflict of interest
The authors declare that there is no conflict of interest regarding the publication of this paper.
This work was partially supported by the National Natural Science Foundation of China under Grant No.61602522 , and the Fundamental Research Funds of the Central Universities of Central South University [No.2018zzts595].
W. William, A. Ware, A. H. Basaza-Ejiri, J. Obungoloch, A review of image analysis and machine learning techniques for automated cervical cancer screening from pap-smear images, Computer methods and programs in biomedicine 164 (2018) 15–22.
- (2) G. A. Mishra, S. A. Pimple, S. S. Shastri, An overview of prevention and early detection of cervical cancers, Indian journal of medical and paediatric oncology: official journal of Indian Society of Medical & Paediatric Oncology 32 (3) (2011) 125.
- (3) K. Bora, M. Chowdhury, L. B. Mahanta, M. K. Kundu, A. K. Das, Automated classification of pap smear images to detect cervical dysplasia, Computer methods and programs in biomedicine 138 (2017) 31–47.
- (4) R. Nayar, D. C. Wilbur, The Bethesda system for reporting cervical cytology: Definitions, criteria, and explanatory notes, Springer, 2015.
- (5) W. William, A. Ware, A. H. Basaza-Ejiri, J. Obungoloch, A pap-smear analysis tool (pat) for detection of cervical cancer from pap-smear images, Biomedical engineering online 18 (1) (2019) 16.
- (6) G. G. Birdsong, Automated screening of cervical cytology specimens, Human pathology 27 (5) (1996) 468–481.
- (7) L. G. Koss, E. Lin, K. Schreiber, P. Elgert, L. Mango, Evaluation of the papnet™ cytologic screening system for quality control of cervical smears, American journal of clinical pathology 101 (2) (1994) 220–229.
- (8) D. C. Wilbur, M. U. Prey, W. M. Miller, G. F. Pawlick, T. J. Colgan, The autopap system for primary screening in cervical cytology. comparing the results of a prospective, intended-use study with routine manual practice., Acta cytologica 42 (1) (1998) 214–220.
- (9) D. C. Wilbur, W. S. Black-Schaffer, R. D. Luff, K. P. Abraham, C. Kemper, J. T. Molina, W. D. Tench, The becton dickinson focalpoint gs imaging system: clinical trials demonstrate significantly improved sensitivity for the detection of important cervical lesions, American journal of clinical pathology 132 (5) (2009) 767–775.
- (10) C. V. Biscotti, A. E. Dawson, B. Dziura, L. Galup, T. Darragh, A. Rahemtulla, L. Wills-Frank, Assisted primary screening using the automated thinprep imaging system, American journal of clinical pathology 123 (2) (2005) 281–287.
- (11) H. C. Kitchener, R. Blanks, G. Dunn, L. Gunn, M. Desai, R. Albrow, J. Mather, D. N. Rana, H. Cubie, C. Moore, et al., Automation-assisted versus manual reading of cervical cytology (mavaric): a randomised controlled trial, The lancet oncology 12 (1) (2011) 56–64.
- (12) E. Bengtsson, P. Malm, Screening for cervical cancer using automated analysis of pap-smears, Computational and mathematical methods in medicine 2014 (2014).
A. GençTav, S. Aksoy, S. ÖNder, Unsupervised segmentation and classification of cervical cell images, Pattern recognition 45 (12) (2012) 4151–4168.
- (14) L. Zhang, H. Kong, C. Ting Chin, S. Liu, X. Fan, T. Wang, S. Chen, Automation-assisted cervical cancer screening in manual liquid-based cytology with hematoxylin and eosin staining, Cytometry Part A 85 (3) (2014) 214–230.
- (15) A. Sarwar, V. Sharma, R. Gupta, Hybrid ensemble learning technique for screening of cervical cancer using papanicolaou smear image analysis, Personalized Medicine Universe 4 (2015) 54–62.
- (16) B. Sharma, K. K. Mangat, Various techniques for classification and segmentation of cervical cell images-a review, International Journal of Computer Applications 147 (9) (2016).
- (17) J. Jantzen, J. Norup, G. Dounias, B. Bjerregaard, Pap-smear benchmark data for pattern classification, Nature inspired Smart Information Systems (NiSIS 2005) (2005) 1–9.
- (18) R. M. Fertig, O. Sangueza, S. Gaudi, A. C. Gamret, J. Cervantes, D. M. Jukic, Whole slide imaging., Am J Dermatopathol (2017) 1.
- (20) J. Redmon, A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767 (2018).
- (21) T. Chankong, N. Theera-Umpon, S. Auephanwiriyakul, Automatic cervical cell segmentation and classification in pap smears, Computer methods and programs in biomedicine 113 (2) (2014) 539–556.
- (22) D. Garcia-Gonzalez, M. Garcia-Silvente, E. Aguirre, A multiscale algorithm for nuclei extraction in pap smear images, Expert Systems with Applications 64 (2016) 512–522.
G. Sun, S. Li, Y. Cao, F. Lang, Cervical cancer diagnosis based on random forest., International Journal of Performability Engineering 13 (4) (2017).
- (24) V. Kudva, K. Prasad, S. Guruvare, Automation of detection of cervical cancer using convolutional neural networks, Critical Reviews™ in Biomedical Engineering 46 (2) (2018).
- (25) M. N. Asiedu, A. Simhal, U. Chaudhary, J. L. Mueller, C. T. Lam, J. W. Schmitt, G. Venegas, G. Sapiro, N. Ramanujam, Development of algorithms for automated detection of cervical pre-cancers with a low-cost, point-of-care, pocket colposcope, IEEE Transactions on Biomedical Engineering (2018).
- (26) P. Bamford, B. Lovell, A water immersion algorithm for cytological image segmentation, in: Proc. APRS Image Segmentation Workshop, Citeseer, 1996, pp. 75–79.
- (27) S. Anantha Sivaprakasam, E. Naganathan, Segmentation and classification of cervical cytology images using morphological and statistical operations., ICTACT Journal on Image & Video Processing 7 (3) (2017).
- (28) M.-H. Tsai, Y.-K. Chan, Z.-Z. Lin, S.-F. Yang-Mao, P.-C. Huang, Nucleus and cytoplast contour detector of cervical smear image, Pattern Recognition Letters 29 (9) (2008) 1441–1453.
- (29) C. Bergmeir, M. G. Silvente, J. M. Benítez, Segmentation of cervical cell nuclei in high-resolution microscopic images: a new algorithm and a web-based software framework, Computer methods and programs in biomedicine 107 (3) (2012) 497–512.
- (30) M. E. Plissiti, C. Nikou, A. Charchanti, Automated detection of cell nuclei in pap smear images using morphological reconstruction and clustering, IEEE Transactions on information technology in biomedicine 15 (2) (2010) 233–241.
- (31) K. Li, Z. Lu, W. Liu, J. Yin, Cytoplasm and nucleus segmentation in cervical smear images using radiating gvf snake, Pattern recognition 45 (4) (2012) 1255–1264.
- (32) T. Guan, D. Zhou, Y. Liu, Accurate segmentation of partially overlapping cervical cells based on dynamic sparse contour searching and gvf snake model, IEEE journal of biomedical and health informatics 19 (4) (2014) 1494–1504.
- (33) D. V. K. Yarlagadda, P. Rao, D. Rao, O. Tawfik, A system for one-shot learning of cervical cancer cell classification in histopathology images, in: Medical Imaging 2019: Digital Pathology, Vol. 10956, International Society for Optics and Photonics, 2019, p. 1095611.
- (34) H. Wieslander, G. Forslid, E. Bengtsson, C. Wahlby, J.-M. Hirsch, C. Runow Stark, S. Kecheril Sadanandan, Deep convolutional neural networks for detecting cellular changes due to malignancy, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 82–89.
- (35) Y. Song, E.-L. Tan, X. Jiang, J.-Z. Cheng, D. Ni, S. Chen, B. Lei, T. Wang, Accurate cervical cell segmentation from overlapping clumps in pap smear images, IEEE transactions on medical imaging 36 (1) (2016) 288–300.
- (36) Z. Lu, G. Carneiro, A. Bradley, D. Ushizima, M. S. Nosrati, A. Bianchi, C. Carneiro, G. Hamarneh, Evaluation of three algorithms for the segmentation of overlapping cervical cells, IEEE J Biomed Health Inform PP (99) (2017) 1–1.
- (37) Z. Lu, G. Carneiro, A. P. Bradley, An improved joint optimization of multiple level set functions for the segmentation of overlapping cervical cells, IEEE Transactions on Image Processing 24 (4) (2015) 1261–1272.
- (38) Z. Lu, G. Carneiro, A. P. Bradley, Automated nucleus and cytoplasm segmentation of overlapping cervical cells, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2013, pp. 452–460.
- (39) M. Arya, N. Mittal, G. Singh, Clustering techniques on pap-smear images for the detection of cervical cancer (2018).
- (40) W. William, A. Ware, A. H. Basaza-Ejiri, J. Obungoloch, Cervical cancer classification from pap-smears using an enhanced fuzzy c-means algorithm, Informatics in Medicine Unlocked 14 (2019) 23–33.
- (41) I. Arganda-Carreras, V. Kaynig, C. Rueden, K. W. Eliceiri, J. Schindelin, A. Cardona, H. Sebastian Seung, Trainable weka segmentation: a machine learning tool for microscopy pixel classification, Bioinformatics 33 (15) (2017) 2424–2426.
- (42) L. Zhang, H. Kong, C. T. Chin, S. Liu, Z. Chen, T. Wang, S. Chen, Segmentation of cytoplasm and nuclei of abnormal cells in cervical cytology using global and local graph cuts, Computerized Medical Imaging and Graphics 38 (5) (2014) 369–380.
- (43) P. Wang, L. Wang, Y. Li, Q. Song, S. Lv, X. Hu, Automatic cell nuclei segmentation and classification of cervical pap smear images, Biomedical Signal Processing and Control 48 (2019) 93–103.
- (44) P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, Y. LeCun, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229 (2013).
- (45) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International journal of computer vision 115 (3) (2015) 211–252.
- (46) R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
- (47) K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE transactions on pattern analysis and machine intelligence 37 (9) (2015) 1904–1916.
- (48) R. Girshick, Fast r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448.
- (49) S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, 2015, pp. 91–99.
- (50) W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: European conference on computer vision, Springer, 2016, pp. 21–37.
- (51) J. Jeong, H. Park, N. Kwak, Enhancement of ssd by concatenating feature maps for object detection, arXiv preprint arXiv:1705.09587 (2017).
- (52) C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, A. C. Berg, Dssd: Deconvolutional single shot detector, arXiv preprint arXiv:1701.06659 (2017).
- (53) Q. Dou, H. Chen, L. Yu, L. Zhao, J. Qin, D. Wang, V. C. Mok, L. Shi, P.-A. Heng, Automatic detection of cerebral microbleeds from mr images via 3d convolutional neural networks, IEEE transactions on medical imaging 35 (5) (2016) 1182–1195.
- (54) A. Teramoto, H. Fujita, O. Yamamuro, T. Tamaki, Automated detection of pulmonary nodules in pet/ct images: Ensemble false-positive reduction using a convolutional neural network technique, Medical physics 43 (6Part1) (2016) 2821–2827.
- (55) Y. Liang, Z. Tang, M. Yan, J. Liu, Object detection based on deep learning for urine sediment examination, Biocybernetics and Biomedical Engineering 38 (3) (2018) 661–670.
- (56) R. Bharath, P. Rajalakshmi, M. A. Mateen, Multi-modal framework for automatic detection of diagnostically important regions in nonalcoholic fatty liver ultrasonic images, Biocybernetics and Biomedical Engineering 38 (3) (2018) 586–601.
- (57) Y. Liang, R. Kang, C. Lian, Y. Mao, An end-to-end system for automatic urinary particle recognition with convolutional neural network, Journal of medical systems 42 (9) (2018) 165.
- (58) M. Wu, C. Yan, H. Liu, Q. Liu, Y. Yin, Automatic classification of cervical cancer from cytological images by using convolutional neural network, Bioscience reports 38 (6) (2018).
- (59) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
- (60) C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
- (61) M. Everingham, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International journal of computer vision 88 (2) (2010) 303–338.
- (62) T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
- (63) K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
- (64) J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al., Speed/accuracy trade-offs for modern convolutional object detectors, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7310–7311.
- (65) T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision, Springer, 2014, pp. 740–755.
- (66) F. Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258.
- (67) A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861 (2017).
- (68) K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).