Accurate 2D soft segmentation of medical image via SoftGAN network

by   Changwei Wang, et al.

Accurate 2D lung nodules segmentation from medical Computed Tomography (CT) images is crucial in medical applications. Most current approaches cannot achieve precise segmentation results that preserving both rich edge details description and smooth transition representations between image regions due to the tininess, complexities, and irregularities of lung nodule shapes. To address this issue, we propose a novel Cascaded Generative Adversarial Network (CasGAN) to cope with CT images super-resolution and segmentation tasks, in which the semantic soft segmentation form on precise lesion representation is introduced for the first time according to our knowledge, and lesion edges can be retained accurately after our segmentation that can promote rapid acquisition of high-quality large-scale annotation data based on RECIST weak supervision information. Extensive experiments validate that our CasGAN outperforms the state-of-the-art methods greatly in segmentation quality, which is also robust on the application of medical images beyond lung nodules. Besides, we provide a challenging lung nodules soft segmentation dataset of medical CT images for further studies.


page 2

page 3

page 4

page 5

page 6

page 7

page 9


3D Matting: A Soft Segmentation Method Applied in Computed Tomography

Three-dimensional (3D) images, such as CT, MRI, and PET, are common in m...

LGAN: Lung Segmentation in CT Scans Using Generative Adversarial Network

Lung segmentation in computerized tomography (CT) images is an important...

CT Image Enhancement Using Stacked Generative Adversarial Networks and Transfer Learning for Lesion Segmentation Improvement

Automated lesion segmentation from computed tomography (CT) is an import...

CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation

Data availability plays a critical role for the performance of deep lear...

Multi-organ Segmentation Network with Adversarial Performance Validator

CT organ segmentation on computed tomography (CT) images becomes a signi...

CIDI-Lung-Seg: A Single-Click Annotation Tool for Automatic Delineation of Lungs from CT Scans

Accurate and fast extraction of lung volumes from computed tomography (C...

1 Introduction

Figure 1: Various types of lung nodules from LIDC, the first line is the original image, and the second line is the binary mask labeled by the doctor. From left to right: isolated nodule, tiny nodule, cavitary nodule, GGO nodule, juxtapleural nodule and calcific nodule.

Accurate 2D lung nodules segmentation from Computed Tomography (CT) scans plays a crucial role in Computer-Aided Diagnosis (CAD) tasks, such as quantitative disease progression, pathology detection, radiomics analysis, surgical assistance. It is a challenging problem because the medical CT is a low dose scan for the safety of the human body, so CT images are usually low resolution and high noise. As shown in Figure 1

, the lung nodule lesions are usually small (5~10 mm), which brings great trouble for the segmentation, especially for the low-resolution images. Direct interpolation of lesions into high-resolution images may lead to the loss of information on many pathological structures. Besides, lung nodule lesions often have irregular shapes, and the binary mask edges may be extremely complex as the GGO nodule (Figure

1(d)). In this case, even manual labeling cannot preserve the details of these edges. Finally, lung lesions are diverse with similar visual characteristics to their surroundings, making the manually labeling for the segmentation tedious and time-consuming, even for experienced doctors. At present, the labeling of many publicly available lung nodule segmentation datasets is not accurate enough, and the labeling of the same lesion by different doctors is also very different.

Based on the above facts, the accurately labeled medical image segmentation datasets are rare in clinical practice. Currently, Response Evaluation Criteria In Solid Tumors (RECIST) is widely adopted in clinical monitoring of cancer patients, and many modern hospitals’ Picture Archiving and Communication Systems (PACS) have stored a large amount of lesion diameter measurement data linked CT images. Deeplesion dataset[36] published images of lesions labeled with RECIST, containing a lot of lung nodule data. Some researchers [5] employed the RECIST long and short axes as the weak supervision information to finish the segmentation task without the artificial pixel-level mask annotation, needing constant manual selection and multiple iterations with a lot of time and efforts.

To obtain accurate lung nodules segmentation, we propose a new method as shown in Figure 3. The contributions are listed as follows:

  • The soft mask is introduced for the first time in the segmentation task of lung nodules according to our knowledge. We verified its advantages and effectiveness through a series of experiments.

  • A soft mask labeling method is given, which can use RECIST annotation or existing binary mask to obtain large-scale data. A new dataset (LNSM) is provided, containing soft segmentation lesions.

  • A new network named Cascaded Generative Adversarial Network(CasGAN) for accurate lung nodules segmentation is proposed, which can cope with CT images super-resolution and Soft mask generation tasks simultaneously. We have performed extensive evaluations on multiple datasets and verified that our method has the state-of-the-art performance.

2 Related work

In clinical routine, manual or semi-manual techniques for lesion segmentation of CT images are applied, which are subjective, operator-dependent and very time-consuming. Therefore, many automatic methods have been proposed to conquer this challenge, especially for lung nodules segmentation, such as morphological operations based method [18], Conditional Random Field (CRF) based method [35], region growth based method [11], energy optimization based method [8] and so on. However, these hand-craft feature-based methods are not widely applied in clinics, due to their speed and robustness on heterogeneous, low-contrast CT images in real-life.

Convolutional Neural Networks (CNN) have gained new attention for medical image analysis and processing. Semantic image segmentation methods based on Full Convolutional Networks (FCN) [24], Mask R-CNN [14], U-Net [31][10] and double-branch residual network (DB-ResNet) [6] were developed for CT image segmentation with highly competitive results compared to traditional methods. Afterwards, Unet++ [38] connected the encoder and decoder through a series of nested convolutional blocks and achieved advanced results.

Although deep learning based methods have largely reduced the need for manual feature extraction in CT image segmentation, days to weeks are still required to manually label the training and testing segmentation data. Recently, deeplesion dataset

[36] published labeled lesions images, containing a lot of lung nodule data. Some weakly supervised semantic segmentation methods [5][22][28] also use the RECIST long and short axes as the weak supervision information to train iterative CNN models and generate more segmentation data. But the challenges in CT image segmentation remain. On one hand, most current CT segmentation algorithms (both traditional methods and deep learning based approaches) and evaluation criteria (such as IOU and Dice) only focus on binary segmentation area, which are not enough to accurately represent lesion details, especially for pixels that may belong partially to more than one segment. These details are essential for high-quality quantitative disease progression and radiomics analysis.

On the other hand, CT image segmentation still suffers from low resolution, high noise, and low contrast problems. Some methods use CNN to improve image quality [19][21][13], which may provide a good solution to solve the problem of lung nodule enlargement. In particular, Generative Adversarial Network (GAN) can also be used to train the CNN model, especially the multi-stage model based on the generation of antagonist network [33], which adopts the stacking GAN method to improve the resolution and contrast of CT successively. However, due to the lack of high-resolution CT images, low-quality images are often generated from the original images via down-sampling.

In sum, most current approaches cannot achieve precise segmentation results that preserving both rich edge details description and smooth transition representations between image regions due to the tininess, complexities, and irregularities of lung nodule shapes. These issues are exactly what we concern about in this paper.

3 Proposed Method

Unlike many natural images, CT images are often noisy and suffer from low resolution. Directly segmenting tiny lung nodules in such images may generate undesirable visual artifacts and edges with low accuracy. Many segmentation masks of open datasets are low quality, which cannot meet the needs of accurate segmentation. Therefore, we introduced soft mask as a new form of segmentation. We propose a novel Cascaded Generative Adversarial Network (CasGAN) to conquer the low dose medical CT images (Figure 3). Firstly, we employ SRGAN [19] to generate a higher resolution image with enhanced boundary and contrast from original CT images. Then, based on the enhanced image and a bilinear interpolation image, we propose a novel Soft mask segmentation GAN(SoftGAN) layer to generate the soft mask. Our soft mask not only has accurate segmentation results and better visualization, but also can be used to improve the results of binary segmentation by softening the binary labels or inputting as prior information.

3.1 Soft Mask for Lung Nodules Segmentation

3.1.1 Definition of Soft Mask:

In medical image processing, lung nodules are usually labeled and segmented in the form of binary mask. However, there are many disadvantages with binary mask labeling for the lung nodules. First, it does not describe the edge details and morphological features of the lung nodules, which cause additional anatomical information about lesion structure in these marginal regions to be ignored. In addition, the labeling is extremely unbalanced since most of the regions are labeled as non-lesion in the binary mask [17], and this imbalance impairs the training effect of the network a lot.

To overcome the weakness of the binary mask, we introduce soft mask labeling strategy for the first time to build CT dataset of lung nodules segmentation that using soft annotation method to preserve more information on the edge of the lesion. Inspired by matting task, we define the soft mask label value of the -th pixel as a linear combination of the lesion label and non-lesion label :


All the of the image can compose as an alpha matte matrix . Then, we take the closed-form solution in closed-form-matting algorithm [20] for extracting the from a CT image.

3.1.2 Labeling of Soft Mask:

Based on our soft mask strategy, we should construct a soft mask segmentation dataset for the training of lung nodules segmentation networks. Manual labeling the image to a soft mask form is tough work, as the edge details cannot be preserved precisely during the interaction process. Here we firstly generate the trimap including the determined foreground, the determined background and the unknown region, and then use it for the close-form-matting base soft mask generation.

Figure 2: Trimaps and masks. (a) lesion images with RECIST labels. (b) trimaps. (c) soft masks. (d) binary masks.

(1)Trimap Generation: Ideally, the trimap should consist of very small unknown regions around the foreground boundary and contain only partial or mixed pixels, and the accuracy of the trimap is one of the important factors affecting the performance of the matting algorithm. However, generating such a precise trimap requires a lot of manual work and is undesirable. For different data, we have designed the following three strategies for generating trimap:

  • Trimap Generation with RECIST annotations: Inspired by [5], we can generate a high-quality trimap by the grabcut segmentation algorithm based on the RECIST long and short axes as the foreground region, and use the grabcut algorithm to initially segment the image. The result obtained by the grabcut algorithm is processed by the image morphology operation to perform erosion and dilation processing. The obtained trimap images are shown in Figure 2(b).

  • Trimap Generation with inconsistent annotations of different doctors: In LIDC dataset, each lesion contains inconsistent binary mask annotations of four doctors. We set the intersection of these masks labeled by multiple doctors as the foreground lesion area. Then, we set the complement of the union of these labeled masks to the non-lesion area, and other areas are uncertainty area.

  • Trimap Generation with binary masks: Using morphology operation process the binary mask according to its area size.The foreground area is obtained through the erosion operation, the non-lesion area is obtained through the dilation, and the uncertainty area is in the middle.

(2)Soft Mask Generation: In general, the matting algorithm is initialized with the image foreground and background regions from the trimap, and produces segmentation using iterative energy minimization. Our practice validates that the closed-form-matting algorithm is suitable for CT images of lung nodules with complex boundaries and burrs. The images processed by the algorithm have better visual effects. After a slight selection and iteration, we select 1500 CT images to build our soft mask dataset of lung nodules (LNSM).

Actually, after obtaining the soft mask, it can be converted into a binary mask by setting an appropriate threshold. As the matting improves the accuracy compared to the method only using grabcut [5], our method only needs half the number of iterations to get the same quality of masks with 20 ms per image (for 256 256 image: trimap 5ms, matting 15ms).

Figure 3: Our CasGAN network structure. High-resolution lesion images () is generated by SRGAN from the original CT image, the image and the original image resized by bilinear interpolation are fed into our SoftGAN to generate accurate soft mask image().

3.2 Soft Segmentation via CasGAN

3.2.1 SRGAN for CT Image Super-resolution:

In original CT images, the lung nodule diameter is usually very tiny (5~10 mm), which is harmful to the accurate segmentation. Direct interpolation of the lesion image may result in loss of pathological structural information. In order to learn the mapping from low-resolution lesion input images () to high-resolution lesion images (), we downsample the lung nodule image of the Deeplesion dataset [36] from 512 512 to 128 128 for training the advanced SRGAN[19], and use our trained SRGAN to generate high-resolution CT images. SRGAN has been successfully used for natural-image super-resolution, producing high-quality images with more visual details and edges compared to their low-resolution counterparts, and our practice validate that SRGAN can improve visual clarity and contrast of lung nodule images, which are useful for subsequent lung nodule segmentation.

Figure 4: The network structure of Soft Mask Segmentation GAN (SoftGAN).The upper network is a segmentor, and the lower network is a discriminator.
Method U-net [31] Segnet [3] Deeplabv3+ [9] Unet++ [38] CasGAN CE-net [12] CasGAN CasGAN
Dice 0.831 0.832 0.854 0.893 0.911 0.913 0.916 0.919
IoU 0.753 0.748 0.764 0.824 0.852 0.858 0.843 0.864
ACC 0.990 0.989 0.989 0.993 0.993 0.995 0.993 0.994
Table 1: Evaluation on LNSM dataset. CasGAN: CasGAN without soft mask; CasGAN: CasGAN without SRGAN.

3.2.2 SoftGAN for CT Image Soft Segmentation:

In order to achieve accurate segmentation of lung nodules by neural networks, the soft mask image can be taken as the ground truth segmentation, and the super-resolution image of lung nodules can be employed to enhance the input CT image. Inspired by SRGAN [19] and SegGAN [37], we introduce the adversarial learning strategy in the classic U-net segmentation network, using a pixel wise discriminator to supervise the segmentation results. Our Soft Mask Segmentation GAN(SoftGAN) layer consists of segmentor () and discriminator()(Figure 3). As shown in Figure 4, in order to preserve more information about the original lesion image, we bilinear interpolate the original image with the same resolution of the high resolution image generated by SRGAN, and concatenate them together into a condition image as the input of SoftGAN, while the output of SoftGAN is the soft mask image . As shown in Figure 4

, our segmentor does not use pooling, and uses LeafyRelu as the activation function in encode, which are the same as DCGAN 


and pix2pix 

[16]. Inspired by lsgan [25], our segmentor uses MSE loss(Equ. 6) minimizing pixel-wise error measurements to supervise the generation of soft masks.


where and are the width and height of original image, while is the amplification factor of to the original image.

Dice Doctor1 Doctor2 Doctor3 Doctor4 50%consensus Ours
Doctor1 1.000 0.706 0.731 0.625 0.775 0.777
Doctor2 0.706 1.000 0.744 0.649 0.829 0.743
Doctor3 0.731 0.744 1.000 0.763 0.830 0.808
Doctor4 0.625 0.649 0.763 1.000 0.697 0.697
50%consensus 0.775 0.829 0.830 0.697 1.000 0.863
Ours 0.777 0.743 0.808 0.697 0.863 1.000
Table 2: The performance of our CasGAN on the LIDC dataset, compared with the results of four doctors and consistency.

Our discriminator uses the structure of CGAN [26] to input the original image and the segmentation result together into the identification. As shown in Figure 4

, our discriminator performs pixel wise discrimination on the segmented soft mask, and the output are the probabilities for all pixels.

Like many methods, our SoftGAN layer add an L1 loss to the segmentor to better optimize the low frequency information, and the total loss is:


3.3 Advantages and Applications of Our Soft Mask

Compared to the binary mask, there are several advantages of our proposed soft mask labeling strategy for lung nodules segmentation:

(1) Soft mask is easier for network learning: For training networks, soft mask labeling is a regularization strategy that softens the traditional one-hot type of label, which can effectively suppress over-fitting while calculating loss values. Specifically, using an independent distribution , the original ground-truth distribution and are mixed and mapped to :


which is consistent with our practice by using the soft mask as ground truth for training. The soft mask softens the binary labels, alleviating sample imbalance and overfitting during training.

For transfer learning, the core idea of Knowledge Distillation 

[15] is to obtain a small model that is more suitable for reasoning from training the large model by migrating knowledge.

In the distillation of large-scale networks [1], Hinton believes that the best training objective function is shown in Equ. 5

, in which the loss function is a combination of the soft and the hard targets, while the weight value of the soft objective function should be larger.


This means that we can use CasGAN to train a soft segmentation network as a teacher network and pass the knowledge to a binary segmentation network to make the student network learn better.

Figure 5: Segmentation network Cascade CasGAN to get prior information.

(2) Soft mask has more accurate segmentation to retain more useful information: There is a lot of pathological information on the edge of the lesion, but the binary mask often does not retain them well. Because soft mask achieves smooth transition of edges, it not only has better visual effects but also represents more edge detail information. As shown in Figure 6, our soft mask contains more information than the binary mask. This information should not be discarded.

Based on these advantages, in addition to better visualization, soft masks can also be used in the following applications:

  • Softening the binary labels: We propose a method to soften the original binary mask image . We take the maximum of the pixels corresponding to and the soft mask obtained by section 3.1. This ensures that the information of the original binary mask is preserved and the edges is smooth. When testing, simply set the threshold to close to 255 to perform binary segmentation. We verified this idea in section 4.2 and 4.3.

  • Cascading CasGAN to provide prior information: As shown in Figure 5, we use CasGAN to generate soft masks as prior information and input them to a segmentation network. This is also a form of soft attention, CasGAN learns to generate a soft mask and uses it to guide the student network to better segmentation. This method can significantly improve some low-quality labeled segmentation tasks. We verified this idea in section 4.3.

  • Radiomics analysis: For radiomics analysis, the ROI (Region Of Interest) of soft mask preserves more edge details of the lung nodules, which is useful for radiomics feature extraction and analysis. We verified this idea in section 4.3.

Figure 6: The line of yellow color in the figure refers to the boundary of the binary mask, and the line of black color refers to the boundary of the soft mask.
Method GC[4] Unet[31]  [27] CF-CNN[34] [32] CasGAN CasGAN CasGAN
Dice 0.689 0.795 0.690 0.822 0.930 0.925 0.925 0.931
Table 3: Comparison with other lung nodule segmentation methods on LIDC. CasGAN: binary mask as ground truth; CasGAN: cascading CasGAN provide soft prior; CasGAN: softening binary mask as ground truth.
Method CasGAN CasGAN CE-net CE-net CasGAN
Dice 0.959 0.868 0.959 0.889 0.966 0.924 0.974 0.976 0.981 0.921
IoU 0.921 0.773 0.922 0.808 0.935 0.862 0.949 0.953 0.963 0.860
ACC 0.927 0.991 0.933 0.992 0.944 0.995 0.957 0.998 0.967 0.995
Table 4: Evaluations on ISBI 2012 (left) and PROMISE 2012 (right). CasGAN: binary mask as ground truth; CasGAN: softening mask as ground truth; CE-net: CE-net cascade CasGAN to get soft prior.
Figure 7: Visual comparison on the LIDC dataset, compared with the results of four doctors and consistency.(a) original image.(b) labeled by doctor1. (c) labeled by doctor2. (d) labeled by doctor3. (e) labeled by doctor4. (f) 50% consensus generated. (g) Our CasGAN generated trained on LNSM.

4 Experiment

In this section, we set multiple experiments on the LNSM, LIDC, PROMISE 2012 [23], and ISBI 2012 dataset [7] to verify the effectiveness of our soft mask and CasGAN.

Evaluation metrics: DICE, IoU, ACC and AUC are used to evaluate our proposed method. For convenient, we use the following denotations: TP for true positive, indicates the number of samples that are actually positive samples predicted to be positive samples; TN for true negative, indicates the number of samples that are actually negative samples predicted to be negative samples; FP for false positive; FN for false negative. Then, we mainly use the above four metrics to evaluate our method as follows. The DICE coefficient is a measure of the amount of uniformity between two image regions and is widely used as a measure for evaluating segmentation performance in medical images, and ; the intersection over union (IoU) score: ; ACC (accuracy rate): ; AUC (area under curve) is a quantitative indicator of the ROC curve.

Training details: As described in section 3.1, we use SRGAN with 5 ResidualBlocks for 4 times CT image super-resolution. The corresponding weights of losses used by the generator of SRGAN are , , and 2e for the content loss (MSE loss), adversarial loss, perception loss, and tv loss respectively. The initial learning rate of the optimizer adam is 0.01, and the iteration is 40 generations. In order to build the training data, high-resolution CT images () are obtained from the Deeplesion, and we downsample these original lesion nodule images to get the low-resolution image (). The network structure of the SoftGAN is shown in Figure 4

corresponding to the segmentor and discriminator respectively, and the optimizer is adam. The learning rate is set to 0.002, and starts to uniformly attenuate after 35 epochs. The weight

of L1 loss in our segmentor is set to . In the following experiments, we trained SRGAN model by using the same parameter settings.

4.1 Experiments on Lung Nodules with Soft Mask (LNSM) Dataset

The DeepLesion dataset [36] consists of 32,735 large numbers of important clinical radiological findings (lesions, tumors, lymph nodes, etc.). These findings were from studies from patients, which were tagged and measured by a physician through RECIST diameters. We use the lung lesions of the Deeplesion dataset to build our dataset (LNSM). According to the method described in Section 3.1.2, we obtained CT lung nodules images with soft mask. We randomly divide the dataset into a training set of 1,000 and a test set of 500. All comparison methods use the same dataset settings.

(1) Comparison with other 2D segmentation networks and ablation study:

Due to the differences of the soft mask and the binary mask, which should not be compared directly, we convert soft mask to binary mask and compare it with the baseline of the 2D medical segmentation task. Common segmentation methods segment lesions by pixel-wise discrimination, so they have no way to train soft masks. We binarize the soft masks with a threshold of 128 into a binary mask. For a fair comparison, we also binarize the soft mask generated by CasGAN with 128 as the threshold and compare it with the binary ground truth on the test set. In this experiment, our method compares the segmentation results with some 2D medical image segmentation networks. U-net, Segnet and Deeplabv3

are common baselines in medical image segmentation, Unet and CE-net [12] are recently proposed end-to-end segmentation networks for medical images. We also performed ablation studys to compare the contribution of each part. The results in Table 1 show that our method has achieved the highest index on the evaluation metrics including Dice, IoU and ACC.

Our GasGAN has achieved relatively good performance on LNSM, which also implies that the generative model based on pixel regression on the lesion segmentation task can achieve the same effect as the discrimination model based on classification. According to the results, soft mask significantly improves the model, but SRGAN is not so significant.

(2) Comparison with doctors: To further verify the segmentation performance and generalization capabilities of our CasGAN, we tested our CasGAN model on the LIDC dataset with doctors, which is trained on our LNSM dataset. We selected 500 lung nodules from LIDC , and use the model not retrained on LIDC to compare with the mask labeled by the 4 doctors and doctors’ 50% consensus images (merge areas labeled by at least 50% of doctors). As shown in Figure 7 and Table 2, our results have better segmentation visualization results. In addition, although there is a gap between the masks labeled by different doctors, our CasGAN’s segmentation results which binarize with 128 as the threshold are the closest to the doctors’ 50% consensus, with the dice score % (as shown in Table 2). At the same time, the experimental results in Figure 7 can also validate that our CasGAN trained on our LNSM dataset is robust and versatile on different lung nodule datasets.

4.2 Experiments on LIDC dataset

The public dataset Lung Image Database Consortium and Image Database Resource Initiative (LIDC) is created by the Lung Image Database Alliance and the Image Database Resource Program (LIDC-IDRI) [2], which contains 1356 nodules from 1018 cases. Almost every nodule case contains the coordinates of the painting boundary and visual features of four radiologists.

Comparison with other Lung nodule segmentation methods: We selected 119 lesions from the LIDC dataset for train and test, of which 60 were used as training sets and 40 were used as test sets. In this experiment we use the method mentioned in section 3.1.2 to label the soft mask with annotation of four doctors. Specially, we used softening the binary mask strategy to obtain training data in section 3.3. While training CasGAN with a binary mask, we also cascading a trained CasGAN generate soft mask to provide priori information. Our method is compared with the Dice scores claimed by these methods. Note that, graph cut (GC) [4] and Unet [31] are widely used in the field of medical image segmentation, and are used as the baseline method on many datasets. Mukharji et al. [27] proposed a method of lung nodule segmentation using graph cut and deep learning probability map. Roy et al. [32] combined deep learning with shape driven level sets and used a coarse-to-fine solution. CF-CNN [34] proposed a central focused convolutional neural network.

The experimental results in table 3 show that our CasGAN has achieved very good performance on LIDC, and the softened binary mask has also improved significantly. Cascading CasGAN provides the prior is mainly effective on low-quality annotations. We use a selected high-quality mask datasets, so dice score did not improve.

4.3 Other Experiments

(1) Validated on other medical datasets:

To validate our method for the application of medical images beyond lung nodules, we performed experiments on ISBI 2012 and PROMISE 2012 [23]

. ISBI 2012 is the task of segment neuronal structure. We augmented the ISBI dataset to obtain 300 images. We use 240 of them as the training set and 60 as the testing set. PROMISE 2012 is the task of segment prostate MRI volumes. The prostate can assume a variety of appearances in different scans, so this is a difficult task. We used 526 images as training set and 131 as testing set. In the ISBI 2012 and PROMISE 2012 dataset, we using soft segmentation CasGAN provide soft prior for binary segmentation CasGAN and CE-net respectively. The experimental results are shown in the following table

4. Our strategy have performed well on these two datasets, so our method has a good prospect for generalization.

(2) Verified the superiority of soft mask in radiomics: Radiomics has demonstrated the relevance of a variety of quantitative imaging biomarkers for various cancer types. Usually, benign lesions are smooth (regular) and malignant lesions are burrs that can be judged by doctors directly according to the margin, sphericity and spiculation characteristics. After doctors selected, we set up a classification experiment of smoothness and burrs to verify the soft mask’s ability of shape expression. We selected lung nodule images on the LNSM data and on LIDC, with the ratio of smooth to burr is respectively. We extracted features using the professional tool pyradiomic [29]

, and then used common machine learning methods (decision trees, KNN, svm) to classify images based on these features. The soft mask is obtained using the method mentioned in section 3.1. The LIDC binary mask is obtained by using the four doctors’ 50% consistent masks mentioned earlier, and the LNSM binary mask is obtained by using a soft mask with 128 as the threshold. The results are shown in Table 


Dataset Model Binary mask Soft mask
Decision tree 0.81 0.81 0.83 0.81
LNSM KNN 0.77 0.87 0.87 0.94
SVM 0.84 0.90 0.88 0.91
Decision tree 0.683 0.63 0.83 0.88
LIDC KNN 0.65 0.75 0.83 0.92
SVM 0.72 0.76 0.83 0.92
Table 5: The classification accuracy and auc of the radiomics features of soft mask and binary mask are obtained by using common machine learning methods.

We don’t extract the shape features but the radiomics features in the ROI, so both too large and too small ROI will impair the results. The ROI of the soft mask contains more pathological information around the lesion, indicating that the soft mask is more conducive to characterizing lesions than the binary mask, and has positive significance for radiomics feature extraction and subsequent diagnostic analysis. Both soft mask and binary mask have their features extracted from soft mask ROI and binary mask ROI, and their intermediate region ROI can be used at the same time in radiomics analysis.

5 Conclusion

To solve the problem of accurate segmentation of lung nodules, we proposed CasGAN which can cope with CT images super-resolution and segmentation tasks simultaneous to achieve accurate segmentation of lung nodules. In our CasGAN, soft mask is introduced which is conducive to both radiomics and network learning, and can be employed to improve other medical image segmentation tasks. We also proposed methods for generating soft masks using RECIST weak supervisory information or binary mask. Soft mask can improve the performance of binary segmentation by softening the binary mask and providing the prior information. Soft mask can be applied as a trick to rough and low quality labeled challenges and datasets. Besides, we provide a new dataset for studies, containing soft mask images.

Figure 8: Experiments on ISBI 2012 [7] and PROMISE 2012 [23]. From left to right are the original image, the trimap image, the soft mask image obtained by 3.1, the results of our CasGAN prediction, and the ground truth.


  • [1] R. Anil, G. Pereyra, A. Passos, R. Ormándi, G. E. Dahl, and G. E. Hinton (2018) Large scale distributed neural network training through online distillation. CoRR abs/1804.03235. Cited by: §3.3.
  • [2] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, and etc. (2011) The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical Physics 38 (2), pp. 915–931. Cited by: §4.2.
  • [3] V. Badrinarayanan, A. Kendall, and R. Cipolla (2015) SegNet: A deep convolutional encoder-decoder architecture for image segmentation. CoRR abs/1511.00561. Cited by: Table 1.
  • [4] Y. Boykov and V. Kolmogorov (2001) An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In

    International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition

    Cited by: Table 3, §4.2.
  • [5] J. Cai, Y. Tang, L. Lu, A. P. Harrison, K. Yan, J. Xiao, L. Yang, and R. M. Summers (2018) Accurate weakly supervised deep lesion segmentation on CT scans: self-paced 3d mask generation from RECIST. CoRR abs/1801.08614. Cited by: §1, §2, 1st item, §3.1.2.
  • [6] H. Cao, H. Liu, E. Song, C. Hung, G. Ma, X. Xu, R. Jin, and J. Lu (2019) Dual-branch residual network for lung nodule segmentation. CoRR abs/1905.08413. Cited by: §2.
  • [7] I. Challenge Segmentation of neuronal structures in em stacks. Note: Cited by: §4, Figure 8.
  • [8] T. F. Chan and L. A. Vese (2001-02) Active contours without edges. Trans. Img. Proc. 10 (2), pp. 266–277. External Links: ISSN 1057-7149 Cited by: §2.
  • [9] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam (2018-09) Encoder-decoder with atrous separable convolution for semantic image segmentation. In The European Conference on Computer Vision (ECCV), Cited by: Table 1.
  • [10] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger (2016) 3D u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, Cham, pp. 424–432. Cited by: §2.
  • [11] J. Dehmeshki, H. Amin, M. Valdivieso, and X. Ye (2008-04) Segmentation of pulmonary nodules in thoracic ct scans: a region growing approach. IEEE Transactions on Medical Imaging 27 (4), pp. 467–480. External Links: ISSN Cited by: §2.
  • [12] Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, and J. Liu (2019) CE-net: context encoder network for 2d medical image segmentation. IEEE Transactions on Medical Imaging 38 (10), pp. 2281–2292. Cited by: Table 1, §4.1.
  • [13] M. Haris, G. Shakhnarovich, and N. Ukita (2018-06) Deep back-projection networks for super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
  • [14] K. He, G. Gkioxari, P. Dollar, and R. Girshick (2017-10) Mask r-cnn. In The IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
  • [15] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. External Links: 1503.02531 Cited by: §3.3.
  • [16] P. Isola, J. Zhu, T. Zhou, and A. A. Efros (2017-07) Image-to-image translation with conditional adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §3.2.2.
  • [17] E. Kats, J. Goldberger, and H. Greenspan (2019) Soft labeling by distilling anatomical knowledge for improved MS lesion segmentation. CoRR abs/1901.09263. Cited by: §3.1.1.
  • [18] W.J. Kostis, A.P. Reeves, D.F. Yankelevitz, and C.I. Henschke (2003)

    Three-dimensional segmentation and growth-rate estimation of small pulmonary nodules in helical ct images

    IEEE Transactions on Medical Imaging 22 (10), pp. 1259–1274. Cited by: §2.
  • [19] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi (2017-07) Photo-realistic single image super-resolution using a generative adversarial network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2, §3.2.1, §3.2.2, §3.
  • [20] A. Levin, D. Lischinski, and Y. Weiss (2008-03) A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence 30, pp. 228–42. Cited by: §3.1.1.
  • [21] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee (2017-07) Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Cited by: §2.
  • [22] D. Lin, J. Dai, J. Jia, K. He, and J. Sun (2016-06) ScribbleSup: scribble-supervised convolutional networks for semantic segmentation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , pp. 3159–3167. External Links: Document, ISSN Cited by: §2.
  • [23] Litjens Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Medical Image Analysis 18 (2), pp. 359–373. Cited by: §4.3, §4, Figure 8.
  • [24] J. Long, E. Shelhamer, and T. Darrell (2015-06) Fully convolutional networks for semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
  • [25] X. Mao, Q. Li, H. Xie, R. Y.K. Lau, Z. Wang, and S. Paul Smolley (2017-10) Least squares generative adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), Cited by: §3.2.2.
  • [26] M. Mirza and S. Osindero (2014) Conditional generative adversarial nets. CoRR abs/1411.1784. Cited by: §3.2.2.
  • [27] S. Mukherjee, X. Huang, and R. R. Bhagalia (2017) Lung nodule segmentation using deep learned prior based graph cut. In The 2017 IEEE International Symposium on Biomedical Imaging, Cited by: Table 3, §4.2.
  • [28] G. Papandreou, L. Chen, K. P. Murphy, and A. L. Yuille (2015)

    Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation

    In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Washington, DC, USA, pp. 1742–1750. Cited by: §2.
  • [29] pyradiomics Pyradiomics tool. Note: Cited by: §4.3.
  • [30] A. Radford, L. Metz, and S. Chintala (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. Computer Science. Cited by: §3.2.2.
  • [31] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham, pp. 234–241. Cited by: §2, Table 1, Table 3, §4.2.
  • [32] R. Roy, T. Chakraborty, and A. Chowdhury (2019-03) A deep learning-shape driven level set synergism for pulmonary nodule segmentation. Pattern Recognition Letters 123, pp. 31–38. Cited by: Table 3, §4.2.
  • [33] Y. Tang, J. Cai, L. Lu, A. P. Harrison, K. Yan, J. Xiao, L. Yang, and R. M. Summers (2018) CT image enhancement using stacked generative adversarial networks and transfer learning for lesion segmentation improvement. In Machine Learning in Medical Imaging - 9th International Workshop, MLMI 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings, pp. 46–54. Cited by: §2.
  • [34] S. Wang, M. Zhou, Z. Liu, Z. Liu, D. Gu, Y. Zang, D. Dong, O. Gevaert, and J. Tian Central focused convolutional neural networks: developing a data-driven model for lung nodule segmentation. Medical Image Analysis 40, pp. 172–183. Cited by: Table 3, §4.2.
  • [35] D. Wu, L. Lu, J. Bi, Y. Shinagawa, K. Boyer, A. Krishnan, and M. Salganicoff (2010-06) Stratified learning of local anatomical context for lung nodules in ct images. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. , pp. 2791–2798. External Links: ISSN Cited by: §2.
  • [36] K. Yan, X. Wang, L. Lu, and R. M. Summers (2018) DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. Journal of Medical Imaging 5 (3), pp. 1 – 11. Cited by: §1, §2, §3.2.1, §4.1.
  • [37] X. Zhang, X. Zhu, 3. X. Zhang, N. Zhang, P. Li, and L. Wang (2018-Sep.) SegGAN: semantic segmentation with generative adversarial network. In 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), Vol. , pp. 1–5. External Links: Document, ISSN Cited by: §3.2.2.
  • [38] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang (2018) UNet++: A nested u-net architecture for medical image segmentation. CoRR abs/1807.10165. Cited by: §2, Table 1.