UACANet: Uncertainty Augmented Context Attention for Polyp Semgnetaion

by   Taehun Kim, et al.

We propose Uncertainty Augmented Context Attention network (UACANet) for polyp segmentation which consider a uncertain area of the saliency map. We construct a modified version of U-Net shape network with additional encoder and decoder and compute a saliency map in each bottom-up stream prediction module and propagate to the next prediction module. In each prediction module, previously predicted saliency map is utilized to compute foreground, background and uncertain area map and we aggregate the feature map with three area maps for each representation. Then we compute the relation between each representation and each pixel in the feature map. We conduct experiments on five popular polyp segmentation benchmarks, Kvasir, CVC-ClinicDB, ETIS, CVC-ColonDB and CVC-300, and achieve state-of-the-art performance. Especially, we achieve 76.6 to the previous state-of-the-art method.



There are no comments yet.


page 3

page 5

page 6

page 8


SalyPath360: Saliency and Scanpath Prediction Framework for Omnidirectional Images

This paper introduces a new framework to predict visual attention of omn...

Salient Object Detection via Augmented Hypotheses

In this paper, we propose using augmented hypotheses which consider obje...

RecSal : Deep Recursive Supervision for Visual Saliency Prediction

State-of-the-art saliency prediction methods develop upon model architec...

CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation

Nucleus segmentation is a challenging task due to the crowded distributi...

Learning Saliency Prediction From Sparse Fixation Pixel Map

Ground truth for saliency prediction datasets consists of two types of m...

Local Context Attention for Salient Object Segmentation

Salient object segmentation aims at distinguishing various salient objec...

Learning Attraction Field Representation for Robust Line Segment Detection

This paper presents a region-partition based attraction field dual repre...

Code Repositories


Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Image segmentation is one of the fundamental and challenging topics in computer vision. It aims to classify each pixel from a given image. Recent studies adopt image segmentation to the specific domain such as salient object detection which focus on classifying each pixel whether it belongs to the most salient object or not. Similar to the salient object detection, one can apply their techniques to the medical purposes.

Medical image segmentation is widely used technique such as classifying each organ in the given tomography images like pancreas segmentation (Oktay et al., 2018), detecting cells from the microscopy images (Ronneberger et al., 2015), or discriminating abnormal regions from normal regions from the body such as brain tumor (Haghighi et al., 2020) or polyp segmentation (Fan et al., 2020).

Polyps are an abnormal tissue growth from a surface of our body and can be found in colon, rectum, stomach or even throat. In most cases, polyps are benign, which means they are not indicating illness or maliciousness. However, because polyps are potentially cancerous, so we need a long-term diagnosis including the growth of their sizes or location and whether it became malignant or not. Thus, detecting polyps in the given colonoscopy image is beneficial for aiding early diagnosis of polyp related diseases.

Previous polyp segmentation networks usually adopt their methodologies from salient object detection (SOD) since they share the main interest, attend more on salient (polyp) region than surrounding scene. Current state-of-the-art methods in SOD which shows decent performance are highly related to the edge guidance (Yang et al., 2017; Su et al., 2019). However, acquiring additional edge data is often expensive. Reverse attention (Chen et al., 2018) suggest using reverse saliency map to obtain boundary cues, but since the boundary region is highly related to the ambiguous saliency score, saliency map without reverse operation already has such boundary information.

In this paper, we propose Uncertainty Augmented Context Attention network (UACANet), augmenting uncertain area with respect to the saliency map which is highly related to the boundary information. Our method computes the region with ambiguous saliency score and combine with a foreground and a background area for context attention module. On top of a modified version of U-Net (Ronneberger et al., 2015)

structure network with additional encoders and decoder, we aggregate the feature map based on three areas with weighted summation to obtain a representative context vectors for each area. We then compute the similarity between the context vector and the feature map. We validate our method with five famous polyp segmentation benchmarks, Kvasir, CVC-ClinicDB, ETIS, CVC-ColonDB and CVC-300 and achieve

state-of-the-art performance among previous methods.

2. Related Work

2.1. Semantic Segmentation

Fully Convolutional Network (FCN) (Long et al., 2015)

introduced a fully convolutional neural network architecture to efficiently train a model to classify each pixel. They also utilized a multi-scale scheme with aggregating low-level feature maps from the early stages in the neural network. Noh

et al. proposed deconvolution network (Noh et al., 2015) to compensate a spatial information degradation due to the pooling operation by leveraging transposed convolution and unpooling methods. Pyramid Spatial Pooling Network (PSPNet) utilized grid-wise pooling and comprise multiple sizes of the grid to deal with multi-scale objects. Deeplab (Chen et al., 2017) constructed multiple convolution layers with different dilation rates to diversify receptive fields within a single module to extract a multi-scale context information. Dual Attention Network (Fu et al., 2019) brought a self-attention mechanism to the semantic segmentation network and comprise a dual path network for spatial and channel dimensions which incorporates local features with their global dependencies. Object Contextual Representation (OCR) (Yuan et al., 2019) expanded a non-local operation to consider semantic region by aggregating the representation of the pixels from each class and compute a similarity with each pixel from the feature map.

2.2. Salient Object Detection

Salient object detection (SOD) resembles semantic segmentation but has its own criterion. Rather than predicting the entire region in the given image and classifying an object class label, SOD focuses on the object itself, identifying more important regions from the surrounding areas. Since salient object does not have any specific object class, even if the task might seem superficially easier than multi-class segmentation, it is much harder when it comes to the accurate object detection and assuming their priorities among other surrounding objects. Unlike semantic segmentation, it is hard to say that there is a strong baseline model with a high performing yet simple architecture in SOD.

Current state-of-the-art methods exploit the boundary region of objects for complementary information by multi-task learning strategy which can be incorporated to enhance the quality of saliency map. Edge Guidance Network (EGNet) (Yang et al., 2017)

used a bottom-up stream for edge detection branch and side-out fusion strategy which aggregates each path from the top-down stream for salient object branch. To clarify the terms that we use as bottom-up stream means from high-level feature level to low-level feature level from the backbone network which is commonly used for segmentation or heat-map related tasks such as pose estimation. Also, side-out fusion strategy aggregates multiple feature maps from the backbone with concatenation or summation. Boundary-aware Network (BANet)

(Su et al., 2019) oppositely used a side-out fusion for boundary branch and single-stream for object branch, but rather than regarding edge detection as a separate task, they combine edge result and object result to generate saliency map. These methods qualitatively showed competitive results consistently (Wang et al., 2021) which proves that edge guidance helps better representation of objects in terms of edge-preserving results. However, acquiring additional edge dataset is often expensive and even with image processing techniques such as Canny edge detection (Canny, 1986), it contains redundant edges which are usually unrelated to the object.

Reverse attention (Chen et al., 2018) explicitly multiply a reverse area of prediction in order to capture residual details for the saliency refinement. However, based on both their experimental results and our experiments, the performance with or without reverse attention wasn’t very different. However, their suggestion provided an intuition that even without explicit edge guidance, we can access to the edge related context with saliency map. Based on this idea, we broaden a reverse saliency map with additional uncertain area, an ambiguous area that saliency score is neither biased to the foreground nor the background. Also, they suggested an efficient yet practical network architecture. The decoder at the end of the backbone feature map produces a global saliency map, and in the bottom-up stream, only the saliency map from the previous level is used unlike U-Net shape networks concatenated feature maps. The final prediction thus only requires a low-level feature map and saliency map from the previous level as a guidance. While U-Net like networks predicts saliency map at the last bottom-up layer, proposed architecture from Reverse attention (Chen et al., 2018) which computes saliency map at the intermediate level feature map and saliency map from the bottom-up stream only.

Figure 1. Overall architecture of UACANet.

2.3. Polyp Segmentation

Polyp segmentation aims to precisely segment polyps from the given colonoscopy image. One can say that polyp segmentation is a semantic segmentation with binary class problem. While we can adopt such fully convolutional network architecture to solve the problem, since colonoscopy has different image domain compared to the general images, researchers focus on extracting semantic features with detail information. Famous architectures like PSPNet or Deeplab adopt multi-scale scheme to the backbone network which helps to capture detail information with multiple receptive field sizes, but they usually deploy such methods at the end of the backbone network which is spatially insufficient for recovering precise spatial information. DeeplabV3+ (Chen et al., 2017) tried to compensate such problem by concatenate low-level feature map from the backbone network but wasn’t sufficient for recovering such details coming from the input image.

U-Net (Ronneberger et al., 2015) introduced incremental up-sampling of the feature maps alongside the corresponding scales of the low-level feature maps. Such ”skip-connections” also appears in Feature Pyramid Networks (FPN) (Lin et al., 2017) but they use element-wise addition while U-Net aggregate features with channel-wise concatenation. While FPN is also designed similar to the U-Net in order to extract multi-scale features, they do not need a fine-grained detail features since object detection needs to locate a bounding box, not a precise shape of an object. U-Net++ (Zhou et al., 2018) added extra layers and dense connectivity to reduce the gap between low-level and high-level features.

As the significance of polyp segmentation has been increased, studies that only dedicated to the polyp datasets have been done recently. ResUNet++ (Jha et al., 2019) construct a U-Net shape network with famous CNN modules including residual blocks from ResNet, Atrous Spatial Pyramid Pooling (ASPP) module from Deeplab and Squeeze and Excitation mechanism from SENet (Hu et al., 2018). Selective Feature Aggregation network (SFA) (Fang et al., ) added another bottom-up path for boundary estimation, similar to SOD networks with edge guidances (Yang et al., 2017; Su et al., 2019). Parallel Reverse Attention Network (PraNet) (Fan et al., 2020) achieved state-of-the-art performance on five different polyp segmentation benchmark by adopting the majority of the network design and techniques from (Chen et al., 2018). They added parallel partial decoder from (Wu et al., 2019) to combine low-level feature maps from the backbone network because the architecture from (Chen et al., 2018) obviously lack of feature sharing for both top-down stream and bottom-up stream. In our network design, we locate intermediate encoder for low-level features and use the output of encoder and previous decoder features for bottom-up stream to incorporate high-level semantic features.

3. Methodology

In this section, we demonstrate the architecture of our UACANet and the details of comprising modules. We first explain the overall structure of our network, then describe the details of fundamental components including Parallel Axial Attention encoder and decoder, and Uncertainty Augmented Context Attention.

3.1. Overall Architecture

We design UACANet based on the overall architecture from PraNet(Fan et al., 2020) which is a modified version of (Chen et al., 2018). As shown in Figure 1, We add additional encoder network, Parallel Axial Attention encoder (PAA-e) for bottom-up stream and side-out fusion path. This helps to reduce the computational cost for both bottom-up and side-out fusion paths by reducing the number of channels for their input feature maps. Feature maps from three PAA-e modules are both used for side-out fusion path (purple arrow), Parallel Axial Attention decoder (PAA-d) and Uncertainty Augmented Context Attention (UACA). We concatenate three feature maps from PAA-e modules for PAA-d and it predicts the initial saliency map for polyps. Then the feature maps from PAA-e and PAA-d is concatenated for UACA, and the output saliency map from PAA-d is used for context guidance (yellow arrow). We describe detailed information of how UACA incorporates feature maps and context guidance in section 3.3

. Then, the output saliency map from UACA is added with previously computed saliency map from the PAA-d. After the first UACA, we concatenate PAA-e feature map and previous UACA feature map for the next UACA (gray and black arrow with concatenation symbol). Also, the saliency map from the previous UACA is used as a context guidance for the next UACA (yellow arrows). The output of UACA is forwarded to the final point-wise convolution and then added with the previous context guidance for the current saliency map. After three consequent UACAs, the final output is computed with bi-linear up-sampling with scale factor of 4 and sigmoid function.

To sum up, the overall architecture shows that the backbone features are encoded with PAA-e, and encoded features are forwarded to PAA-d for initial saliency map which serves as a initial guidance map, which leads UACA to learn a residual saliency map apart from the initial map. This helps the consequent UACA can focus more on uncertain area like boundaries rather than fairly evident region.

We use both binary cross entropy (BCE) loss and intersection over union (IoU) loss. The loss function

is computed as follows,


where refers to a pixel in the output and ground truth, denotes ground truth and denotes the output. As shown in Figure 1, we add four losses from four prediction from PAA-d and UACA with loss function in Equation 1 (red arrows in Figure 1).

Figure 2. The details of Parallel Axial Attention (PAA)

3.2. Parallel Axial Attention encoder and decoder

In the era of deep learning, computer vision researchers who study on semantic segmentation and its related tasks like polyp segmentation are dedicated to find a better architecture to extract a fine-grained feature maps which have both high-level semantic information and low-level detail information because usually they are both hard to extract and to combine each other. Self-attention mechanism

(Zhang et al., 2019) is one of the major context modules in computer vision but requires heavy computation. Axial attention (Ho et al., 2019) solve this problem with performing non-local operation with respect to the single axis and sequentially connected each operation.

We propose Parallel Axial Attention (PAA) for extracting both global dependencies and local representation. We adopt axial attention strategy by computing non-local operation for both horizontal axis and vertical axis but collocated in parallel. By locating vertical and horizontal attention, both contributes to the final output almost equally compared to the sequential method. While sequentially connected axial attention added trainable positional encoding, we do not use encoding scheme since positional encoding substantially not effective in relatively small scales. Also, we discover that using a parallel connection, element-wise summation is effective to aggregate feature maps rather than concatenation without performance degradation since the same input is used for both horizontal axis and vertical axis and they contributes to the output almost equally by the parallel connection. Also, since a single axis based attention cause unexpected deformation, element-wise summation can help to compensate such artifact. As shown in Figure 2, we compute two non-local operations with input feature map, one for horizontal axis and the other for vertical axis.

We choose to actively use this module for encoder and decoder modules for better representation in terms of globally refined feature. First, we design Parallel Axial Attention encoder (PAA-e) which aggregates the low-level feature maps from the top-down stream which will be used for bottom-up stream. Since U-Net structure use the low-level features without channel reduction, redundant information may hinder the performance and the number of channels are quite large since backbone networks are trained for image classification. Not to lose any detail information while reducing the number of channels, we design PAA-e with Receptive Field Block (RFB) (Liu et al., 2018) strategy. As shown in Figure 3(a), feature map from the backbone network (green box) is forwarded to each receptive field path. We add PAA for additional global refinement for each scale and concatenate the outputs, then forward to the consecutive convolution layers. As shown in Figure 1, the outputs of PAA-e are used for both decoder module and bottom-up stream. Also, we design Parallel Axial Attention decoder (PAA-d) with simple structure yet add additional PAA for final feature aggregation from different level of PAA-e features which are denoted as purple arrows in Figure 1 and Figure 3(b).

Figure 3. The details of Parallel Axial Attention encoder (PAA-e) (a) and decoder (PAA-d) (b)

3.3. Uncertainty Augmented Context Attention

While reverse attention (Chen et al., 2018) in both SOD and polyp segmentation does not bring the large margin of performance gain, it was evident that it has shown some qualitatively better results. This phenomenon is highly related to the boundary guided SOD networks (Yang et al., 2017; Su et al., 2019), which show state-of-the-art performances in multiple SOD benchmarks. Boundaries of the object as a extra supervision in SOD networks tends to compensate false negative areas, in other words, object regions with low saliency scores. Reverse attention is thus potentially effective method to bring implicit edge guidance without explicit shape of boundary supervision.

Figure 4. The details of Uncertainty Augmented Context Attention (UACA) module.

We focus on both saliency and reverse saliency map from the reverse attention and found that usually the boundary region appears where saliency score shows ambiguity. In other words, boundary region is highly related to the saliency score around . From this property of saliency map, we assume that both saliency and reverse saliency map has almost equal amount of edge information since reverse saliency map is obtained by simple subtraction from 1. Based on this assumption, extracting ambiguous region from the saliency map as well as foreground and background area would improve attentive methods such as self-attention.

Based on our research, we propose Uncertainty Augmented Context Attention (UACA) module, a novel self-attention mechanism which incorporates uncertain area

for rich semantic feature extraction without extra boundary guidance. We denote previously computed input saliency map as

m and generate corresponding foreground map , background map and uncertain area map as follows,


We compute foreground and background map with max operation in order to disentangle not only from each other but from the uncertain area map since uncertain area map already represents their joint region which makes a redundant information and may diminish the role of uncertainty.

While reverse attention applies explicit channel-wise multiplication to the feature map which resembles Convolutional Block Attention Module (CBAM) (Woo et al., 2018), we compose our context module with a set of non-local operations. Similar to OCR (Yuan et al., 2019) we first compute representative vectors for foreground map, background map and uncertain area map by aggregating the pixel representation with each area map from the input feature map x as follows,


where denotes pixels in spatial dimension. We implement Equation 2 with matrix multiplication as shown in Figure 4. Each vector stands for the representative feature vector, so represents foreground feature and represents the uncertain area. Then we compute the similarity between each representation vector (, and ) and each pixel from the input feature map as follows,



Dataset Method Mean Dice Mean IoU MAE
UACANet-S (w/o PAA) 0.902 0.858 0.008
UACANet-S 0.916 0.870 0.008


ETIS UACANet-S (w/o PAA) 0.684 0.603 0.029
UACANet-S 0.694 0.615 0.023


Table 1. Ablation study for Parallel Axial Attention (PAA) on CVC-ClinicDB and ETIS datasets. denotes higher the better and denotes lower the better.


Dataset Method Mean Dice Mean IoU MAE
CANet-S 0.911 0.857 0.009
CANet-L 0.912 0.861 0.009
UACANet-S 0.916 0.870 0.008
UACANet-L 0.926 0.880 0.006


ETIS CANet-S 0.691 0.613 0.026
CANet-L 0.678 0.604 0.019
UACANet-S 0.694 0.615 0.023
UACANet-L 0.766 0.689 0.012


Table 2. Ablation study for uncertain area on CVC-ClinicDB and ETIS datasets. Red color denotes the best score among the methods, and blue color denotes the second best. denotes higher the better and denotes lower the better.

Finally, we compute context feature map by weighted summation of representation vector , and by similarity score , and as follows,

Figure 5. Qualitative results of comparison with CANet-L and UACANet-L

Note that , , and are point-wise convolution. Each pixel in context feature map, , can be interpreted as a weighted average of three representation vectors , and . The context feature map t and input feature map x are concatenated with respect to the channel axis and feed forward to the point-wise convolution for final output feature map as shown in Figure 4.


Dataset Method Mean Dice Mean IoU MAE
Kvasir U-Net (Ronneberger et al., 2015) 0.818 0.746 0.055
U-Net++ (Zhou et al., 2018) 0.821 0.743 0.048
ResUNet++ (Jha et al., 2019) 0.813 0.793 -
SFA (Fang et al., ) 0.723 0.611 0.075
PraNet (Fan et al., 2020) 0.898 0.840 0.030
UACANet-S (Ours) 0.905 0.852 0.026
UACANet-L (Ours) 0.912 0.859 0.025


U-Net (Ronneberger et al., 2015) 0.823 0.755 0.019
U-Net++ (Zhou et al., 2018) 0.794 0.729 0.022
ResUNet++ (Jha et al., 2019) 0.796 0.796 -
SFA (Fang et al., ) 0.700 0.607 0.042
PraNet (Fan et al., 2020) 0.899 0.849 0.009
UACANet-S (Ours) 0.916 0.870 0.008
UACANet-L (Ours) 0.926 0.880 0.006


Table 3. Comparison to the previous state-of-the-art methods and our UACANet on Kvasir and CVC-ClinicDB datasets. Red color denotes the best score among the methods, and blue color denotes the second best. denotes higher the better and denotes lower the better.

4. Experimental Results

In this section, we demonstrate our implementation details, datasets and benchmarks for experiments, and some experimental results including ablation study on uncertain area and comparison with previous state-of-the-art methods on five polyp segmentation benchmarks. We also visualize feature maps and the uncertainty map from UACA and some qualitative results to compare other methods.


Method ETIS CVC-ColonDB CVC-300
mDice mIoU MAE mDice mIoU MAE mDice mIoU MAE
U-Net (Ronneberger et al., 2015) 0.398 0.335 0.036 0.512 0.444 0.061 0.710 0.627 0.022
U-Net++ (Zhou et al., 2018) 0.401 0.344 0.035 0.483 0.410 0.064 0.707 0.624 0.018
SFA (Fang et al., ) 0.297 0.217 0.109 0.469 0.347 0.094 0.467 0.329 0.065
PraNet (Fan et al., 2020) 0.628 0.567 0.031 0.709 0.640 0.045 0.871 0.797 0.010
UACANet-S (Ours) 0.694 0.615 0.023 0.783 0.704 0.034 0.902 0.837 0.006
UACANet-L (Ours) 0.766 0.689 0.012 0.751 0.678 0.039 0.910 0.849 0.005


Table 4. Comparison to the previous state-of-the-art methods and our UACANet on ETIS, CVC-ColonDB and CVC-300 datasets. Red color denotes the best score among the methods, and blue color denotes the second best. denotes higher the better and denotes lower the better.

4.1. Implementation Details

We describe most of our model architecture description in Section 3 and the number of channels in convolution layers which appear outside of the backbone network is unified as 32 for small model and 256 for large model. We denote the small model with 32 channels as UACANet-S and the large model with 256 channels as UACANet-L. We use Res2Net (Gao et al., 2019) with settings as a backbone network. Intermediate backbone feature maps are acquired from each stage’s last residual block (green box in Figure 1. For UACANet-L, similar to the DeeplabV3+ (Chen et al., 2017)

, we modified strides and dilation rates to increase the spatial size of the feature map. We resize images to

for both training and inference and resize back to its original size. Unlike PraNet (Fan et al., 2020) and other state-of-the-art methods, we adopt additional data augmentation techniques which are fairly popular in semantic segmentation including random flipping on both horizontal and vertical axis, random image scaling from to . We conduct random rotation from to degrees since colonoscopy images may rotate during examination. We also add additional random dilation and erosion for ground truth label to enhance generalization. We use Adam optimizer (Kingma and Ba, 2014) and the initial learning rate is set to with polynomial learning rate decay (Chen et al., 2017) with factor

. Compared to PraNet which trained only 60 epoches, we increase training epoch to 240 since we apply diverse data augmentation. We use Pytorch

(Paszke et al., 2019) to implement our model and single Titan RTX GPU for train the model.

4.2. Datasets

Following (Fan et al., 2020), images which have been randomly selected from Kvasir and CVC-ClinicDB is used for training, but we use same training data for fair comparison which has already been extracted from Kvasir and CVC-ClinicDB and it contains 1450 images total. For benchmark dataset, we use five different datasets.



CVC-ClinicDB (Bernal et al., 2015), also known as CVC-612 contains 612 images from 25 colonoscopy videos and selected 29 sequences from them. The size of images is . 62 images from this dataset are used for test and remaining images are used for training.


CVC-300 is a test dataset from EndoScene (Vázquez et al., 2017). EndoScene contains 912 images from 44 colonoscopy sequences which were acquired from 36 patients total. Since EndoScene dataset is a combination of CVC-ClinicDB and CVC-300, following D.-P. Fan et al, we use CVC-300 as a test dataset which are 60 samples total.


CVC-ColonDB (Bernal et al., 2012) dataset is collected from 15 different colonoscopy sequences and sampled 380 images from these sequences.


ETIS (Silva et al., 2013) dataset contains 196 images which are collected from 34 colonoscopy videos. The size of images is which is the largest among other datasets. Unless polyps in this dataset are vary in size and shape, they are mostly small and hard to find, which makes this dataset more challenging.


Kvasir (Jha et al., 2020) Kvasir dataset consists of 1000 polyp images and corresponding annotations. Unlike the other datasets, images vary in size, from to and also the size of polyps which appear in the images vary in its size and shape. There are 700 large polyps which is larger than , 48 small polyps smaller than and 323 medium polyps within the large and small scale. 900 images are used for training and 100 images are used for test.

4.3. Ablation Study on Parallel Axial Attention

To exhibit the validity of PAA module, we conduct an experiment to evaluate UACANet without PAA modules. We design another model whose specific model architecture is identical to the UACANet except PAA modules (yellow box in Figure 3) is excluded. We choose three metrics to evaluate our methods, mean Dice (mDice), mean intersection over union (mIoU) and mean absolute error (MAE). We choose these two datasets for ablation study since CVC-ClinicDB is sampled for training while ETIS isn’t. As shown in Table 1, UACANet-S with PAA module shows better.

4.4. Ablation Study on uncertain area

We conduct an experiment to demonstrate the effectiveness of UACA. We leave all the details identical to the UACA except for excluding the uncertainty map ( in Figure 4), namely Context Attention (CA). We substitute UACA with CA in Figure 1 to make CANet. We design CANet with same small and large version, namely CANet-S and CANet-L respectively. Table 2 shows quantitative results on CANet and UACANet on CVC-ClinicDB and ETIS. We also choose these two datasets for same reason as section 4.3. In terms of performance measure, UACANet consistently outperforms CANet on three major metrics.

Figure 6. Qualitative results comparison with state-of-the-art methods on five different benchmarks.

We also visualize the output feature map of attention modules in both settings in Figure 5 to verify the effectiveness of uncertain area qualitatively. UACANet substantially produce more precise results than CANet in terms of quality as well. In first and third row, while visualized feature map and the output of CANet-L shows that even though it detects substantial region of polyps, it fails to detect the ambiguous region which may seems hard to discriminate from mucosa, surface of colon. On the other hand, UACANet consistently segment polyp regions precisely. We also visualize uncertain area of UACA ( in Figure 4), it is easy to recognize that uncertain area is closely related to the boundary of polyps. Especially for the third row, also helps to detect the ambiguous area denoted as red boxes. Also, in second row, since ground truth miss another polyp on left bottom corner, UACANet detected the missing polyp.

4.5. Experiments with State-of-the-Art methods

As mentioned above, we compare our method with previous state-of-the-art methods on five different polyp segmentation benchmarks. Since we train on sampled data from Kvasir and CVC-ClinicDB, even the test results are still unseen, the domain of images from these two datasets is still similar to the training data. Thus, we first demonstrate our results on first two datasets on Table 3. On both datasets, our UACANet achieve the best performance among other methods. Especially, our UACANet-L achieves 92.6% mean Dice on CVC-Clinic DB which is 2.7% improvement over PraNet, the latest state-of-the-art method. In Table 4, we evaluate our method with three completely unseen datasets. We mentioned above that ETIS is the most challenging datasets among other four, nevertheless UACANet-L achieve 76.6% mean Dice which is 13.8% improvement over PraNet.

We also demonstrate qualitative results on five benchmarks of previous state-of-the-art methods and our method (Figure 5). On Kvasir and CVC-ClinicDB (first and second row), since two datasets are similar to the training dataset, all methods are able to segment the location of polyps, but our method show the most similar results compared to the ground truth. On ETIS dataset (third row), which is the most challenging dataset among other four benchmarks, both UACANet-S and UACANet-L are able to detect a small polyp even if the size of the polyp is very small and hard to notice while other methods has failed to detect.

5. Conclusion

We propose a novel polyp segmentation network called UACANet which augments uncertain area to the context representation for accurate polyp detection. Without expensive edge annotations, we show that uncertain area is capable of representing boundary information. We propose Parallel Axial Attention for encoder for backbone features and decoder for initial saliency map. We also propose Uncertainty Augmented Context Attention which augments uncertainty area which represents complementary edge information. In a series of both quantitative and qualitative experiments shows that our method outperforms compared to the previous state-of-the-art methods.


This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2017-0-00897, Development of Object Detection and Recognition for Intelligent Vehicles) and (No.2018-0-01290, Development of an Open Dataset and Cognitive Processing Technology for the Recognition of Features Derived From Unstructured Human Motions Used in Self-driving Cars)


  • J. Bernal, J. Sánchez, and F. Vilariño (2012) Towards automatic polyp detection with a polyp appearance model. Pattern Recognition 45 (9), pp. 3166–3182. Note: Best Papers of Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA’2011) External Links: ISSN 0031-3203, Document, Link Cited by: item CVC-ColonDB.
  • J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilariño (2015) WM-dova maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics 43, pp. 99–111. External Links: ISSN 0895-6111, Document, Link Cited by: item CVC-ClinicDB.
  • J. Canny (1986) A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8 (6), pp. 679–698. External Links: Document Cited by: §2.2.
  • L. Chen, G. Papandreou, F. Schroff, and H. Adam (2017) Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587. Cited by: §2.1, §2.3, §4.1.
  • S. Chen, X. Tan, B. Wang, and X. Hu (2018) Reverse attention for salient object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 234–250. Cited by: §1, §2.2, §2.3, §3.1, §3.3.
  • D. Fan, G. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao (2020) Pranet: parallel reverse attention network for polyp segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 263–273. Cited by: §1, §2.3, §3.1, Table 3, §4.1, §4.2, Table 4.
  • [7] Y. Fang, C. Chen, Y. Yuan, and K. Tong Selective feature aggregation network with area-boundary constraints for polyp segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2019Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern RecognitionInternational conference on machine learningProceedings of the European Conference on Computer Vision (ECCV)Proceedings of the European conference on computer vision (ECCV), Berlin, Heidelberg. External Links: ISBN 978-3-030-32238-0, Link, Document Cited by: §2.3, Table 3, Table 4.
  • J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu (2019) Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146–3154. Cited by: §2.1.
  • S. Gao, M. Cheng, K. Zhao, X. Zhang, M. Yang, and P. H. Torr (2019) Res2net: a new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence. Cited by: §4.1.
  • F. Haghighi, M. R. H. Taher, Z. Zhou, M. B. Gotway, and J. Liang (2020) Learning semantics-enriched representation via self-discovery, self-classification, and self-restoration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 137–147. Cited by: §1.
  • J. Ho, N. Kalchbrenner, D. Weissenborn, and T. Salimans (2019) Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180. Cited by: §3.2.
  • J. Hu, L. Shen, and G. Sun (2018) Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141. Cited by: §2.3.
  • D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, and H. D. Johansen (2020) Kvasir-seg: a segmented polyp dataset. In International Conference on Multimedia Modeling, pp. 451–462. Cited by: item Kvasir.
  • D. Jha, P. H. Smedsrud, M. A. Riegler, D. Johansen, T. De Lange, P. Halvorsen, and H. D. Johansen (2019) Resunet++: an advanced architecture for medical image segmentation. In 2019 IEEE International Symposium on Multimedia (ISM), pp. 225–2255. Cited by: §2.3, Table 3.
  • D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.1.
  • T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie (2017) Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125. Cited by: §2.3.
  • S. Liu, D. Huang, et al. (2018) Receptive field block net for accurate and fast object detection. pp. 385–400. Cited by: §3.2.
  • J. Long, E. Shelhamer, and T. Darrell (2015) Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440. Cited by: §2.1.
  • H. Noh, S. Hong, and B. Han (2015) Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pp. 1520–1528. Cited by: §2.1.
  • O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, et al. (2018) Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999. Cited by: §1.
  • A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703. Cited by: §4.1.
  • O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Cited by: §1, §1, §2.3, Table 3, Table 4.
  • J. Silva, A. Histace, O. Romain, X. Dray, and B. Granado (2013) Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. International Journal of Computer Assisted Radiology and Surgery 9 (2), pp. 283–293. External Links: Document, Link Cited by: item ETIS.
  • J. Su, J. Li, Y. Zhang, C. Xia, and Y. Tian (2019) Selectivity or invariance: boundary-aware salient object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3799–3808. Cited by: §1, §2.2, §2.3, §3.3.
  • D. Vázquez, J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, A. M. López, A. Romero, M. Drozdzal, and A. Courville (2017) A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of healthcare engineering 2017. Cited by: item CVC-300.
  • W. Wang, Q. Lai, H. Fu, J. Shen, H. Ling, and R. Yang (2021) Salient object detection in the deep learning era: an in-depth survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2.2.
  • S. Woo, J. Park, J. Lee, and I. S. Kweon (2018) Cbam: convolutional block attention module. pp. 3–19. Cited by: §3.3.
  • Z. Wu, L. Su, and Q. Huang (2019) Cascaded partial decoder for fast and accurate salient object detection. pp. 3907–3916. Cited by: §2.3.
  • B. Yang, X. Zhang, L. Chen, H. Yang, and Z. Gao (2017) Edge guided salient object detection. Neurocomputing 221, pp. 60–71. Cited by: §1, §2.2, §2.3, §3.3.
  • Y. Yuan, X. Chen, and J. Wang (2019) Object-contextual representations for semantic segmentation. arXiv preprint arXiv:1909.11065. Cited by: §2.1, §3.3.
  • H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena (2019)

    Self-attention generative adversarial networks

    pp. 7354–7363. Cited by: §3.2.
  • Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang (2018) Unet++: a nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11. Cited by: §2.3, Table 3, Table 4.