Instance-aware Semantic Segmentation via Multi-task Network Cascades
The topic of semantic segmentation has witnessed considerable progress due to the powerful features learned by convolutional neural networks (CNNs). The current leading approaches for semantic segmentation exploit shape information by extracting CNN features from masked image regions. This strategy introduces artificial boundaries on the images and may impact the quality of the extracted features. Besides, the operations on the raw image domain require to compute thousands of networks on a single image, which is time-consuming. In this paper, we propose to exploit shape information via masking convolutional features. The proposal segments (e.g., super-pixels) are treated as masks on the convolutional feature maps. The CNN features of segments are directly masked out from these maps and used to train classifiers for recognition. We further propose a joint method to handle objects and "stuff" (e.g., grass, sky, water) in the same framework. State-of-the-art results are demonstrated on benchmarks of PASCAL VOC and new PASCAL-CONTEXT, with a compelling computational speed.READ FULL TEXT VIEW PDF
Large-scale data is of crucial importance for learning semantic segmenta...
Recently, significant improvement has been made on semantic object
In this paper we introduce a novel method for general semantic segmentat...
In this paper we introduce a novel method for segmentation that can bene...
We present a two-module approach to semantic segmentation that incorpora...
Despite the initial belief that Convolutional Neural Networks (CNNs) are...
A major challenge in clinical In-Vitro Fertilization (IVF) is selecting ...
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Semantic segmentation [14, 19, 24, 2] aims to label each image pixel to a semantic category. With the recent breakthroughs  by convolutional neural networks (CNNs) , R-CNN based methods [8, 10] for semantic segmentation have substantially advanced the state of the art.
The R-CNN methods [8, 10] for semantic segmentation extract two types of CNN features - one is region features  extracted from proposal bounding boxes ; the other is segment features extracted from the raw image content masked by the segments . The concatenation of these features are used to train classifiers . These methods have demonstrated compelling results on this long-standing challenging task.
However, the raw-image-based R-CNN methods [8, 10] have two issues. First, the masks on the image content can lead to artificial boundaries. These boundaries do not exhibit on the samples during the network pre-training (e.g
., in the 1000-category ImageNet). This issue may degrade the quality of the extracted segment features. Second, similar to the R-CNN method for object detection , these methods need to apply the network on thousands of raw image regions with/without the masks. This is very time-consuming even on high-end GPUs.
The second issue also exists in R-CNN based object detection. Fortunately, this issue can be largely addressed by a recent method called SPP-Net , which computes convolutional feature maps on the entire image only once and applies a spatial pyramid pooling (SPP) strategy to form cropped features for classification. The detection results via these cropped features have shown competitive detection accuracy , and the speed can be faster. Therefore, in this paper, we raise a question: for semantic segmentation, can we use the convolutional feature maps only?
The first part of this work says yes to this question. We design a convolutional feature masking (CFM) method to extract segment features directly from feature maps instead of raw images. With the segments given by the region proposal methods (e.g., selective search ), we project them to the domain of the last convolutional feature maps. The projected segments play as binary functions for masking the convolutional features. The masked features are then fed into the fully-connected layers for recognition. Because the convolutional features are computed from the unmasked image, their quality is not impacted. Besides, this method is efficient as the convolutional feature maps only need to be computed once. The aforementioned two issues involving semantic segmentation are thus both addressed. Figure 1 compares the raw-image-based pipeline and our feature-map-based pipeline.
The second part of this paper further generalizes our method for joint object and stuff segmentation . Different from objects, “stuff”  (e.g., sky, grass, water) is usually treated as the context in the image. Stuff mostly exhibits as colors or textures and has less well-defined shapes. It is thus inappropriate to use a single rectangular box or a single segment to represent stuff. Based on our masked convolutional features, we propose a training procedure that treats a stuff as a compact combination of multiple segment features. This allows us to address the object and stuff in the same framework.
Based on the above methods, we show state-of-the-art results on the PASCAL VOC 2012 benchmark  for object segmentation. Our method can process an image in a fraction of a second, which is faster than the R-CNN-based SDS method . Further, our method is also the firstdeep-learning-based method ever applied to the newly labeled PASCAL-CONTEXT benchmark  for both object and stuff segmentation, where our result substantially outperforms previous states of the art.
The power of CNNs as a generic feature extractor has been gradually revealed in the computer vision area[13, 6, 25, 8, 11]. In Krizhevsky et al.’s work , they suggest that the features of the fully-connected layers can be used as holistic image features, e.g
., for image retrieval. In[6, 25]
, these holistic features are used as generic features for full-image classification tasks in other datasets via transfer learning. In the breakthrough object detection paper of R-CNN, the CNN features are also used like holistic features, but are extracted from sub-images which are the crops of raw images. In the CNN-based semantic segmentation paper , the R-CNN idea is generalized to masked raw image regions. For all these methods, the entire network is treated as a holistic feature extractor, either on the entire image or on sub-images.
In the recent work of SPP-Net , it shows that the convolutional feature maps can be used as localized features. On a full-image convolutional feature map, the local rectangular regions encode both the semantic information (by strengths of activations) and spatial information (by positions). The features from these local regions can be pooled  directly for recognition.
The spatial pyramid pooling (SPP) in  actually plays two roles: 1) masking the feature maps by a rectangular region, outside which the activations are removed; 2) generating a fixed-length feature from this arbitrary sized region. So, if masking by rectangles can be effective, what if we mask the feature maps by a fine segment with an irregular shape?
The Convolutional Feature Masking (CFM) layer is thus developed. We first obtain the candidate segments (like super-pixels) on the raw image. Many regional proposal methods (e.g., [22, 1]) are based on super-pixels. Each proposal box is given by grouping a few super-pixels. We call such a group as a segment proposal. So we can obtain the candidate segments together with their proposal boxes (referred to as “regions” in this paper) without extra effort. These segments are binary masks on the raw images.
Next we project these binary masks to the domain of the last convolutional feature maps. Because each activation in the convolutional feature maps is contributed by a receptive field in the image domain, we first project each activation onto the image domain as the center of its receptive field (following the details in ). Each pixel in the binary masks on the image is assigned to its nearest center of the receptive fields. Then these pixels are projected back onto the convolutional feature map domain based on this center and its activation’s position. On the feature map, each position will collect multiple pixels projected from a binary mask. These binary values are then averaged and thresholded (by 0.5). This gives us a mask on the feature maps (Figure 2). This mask is then applied on the convolutional feature maps. Actually, we only need to multiply this binary mask on each channel of the feature maps. We call the resulting features as segment features in our method.
In , it has been shown that the segment features alone are insufficient. These segment features should be used together with the regional features (from bounding boxes) generated in a way like R-CNN . Based on our CFM layer, we can have two possible ways of doing this.
Design A: on the last convolutional layer. As shown in Figure 3 (left part), after the last convolutional layer, we generate two sources of features. One is the regional feature produced by the SPP layer as in . The other is the segment feature produced in the following way. The CFM layer is applied on the full-image convolutional feature map. This gives us an arbitrary-sized (in terms of its bounding box) segment feature. Then we use another SPP layer on this feature to produce a fixed-length output. The two pooled features are fed into two separate fc layers. The features of the last fc layers are concatenated to train a classifier, as is the classifier in .
In this design, we have two pathways of the fc layers in both training and testing.
Design B: on the spatial pyramid pooling layer. We first adopt the SPP layer  to pool the features. We use a 4-level pyramid of as in . The level is actually a tiny feature map that still has plenty spatial information. We apply the CFM layer on this tiny feature map to produce the segment feature. This feature is then concatenated with the other three levels and fed onto the fc layers, as shown in Figure 3 (right).
In this design, we keep one pathway of the fc layers to reduce the computational cost and over-fitting risk.
Based on these two designs and the CFM layer, the training and inference stages can be easily conducted following the common practices in [8, 11, 10]. In both stages, we use the region proposal algorithm (e.g., selective search ) to generate about 2,000 region proposals and associated segments. The input image is resized to multiple scales (the shorter edge ) , and the convolutional feature maps are extracted from full images and then fixed (not further tuned).
Training. We first apply the SPP method 111https://github.com/ShaoqingRen/SPP_net to finetune a network for object detection. Then we replace the finetuned network with the architecture as in Design A or B, and further finetune the network for segmentation. In the second fine-tuning step, the segment proposal overlapping a ground-truth foreground segment by is considered as positive, and as negative. The overlap is measured by intersection-over-union (IoU) score based on the two segments’ areas (rather than their bounding boxes). After fine-tuning, we train a linear SVM classifier on the network output, for each category. In the SVM training, only the ground-truth segments are used as positive samples.
Inference. Each region proposal is assigned to a proper scale as in . The features of each region and its associated segment are extracted as in Design A or B. The SVM classifier is used to score each region.
Given all the scored region proposals, we obtain the pixel-level category labeling by the pasting scheme in SDS . This pasting scheme sequentially selects the region proposal with the highest score, performs region refinement, inhibits overlapping proposals, and pastes the pixel labels onto the labeling result. Region refinement improves the accuracy by about 1% on PASCAL VOC 2012 for both SDS and our method.
We evaluate our method on the PASCAL VOC 2012 semantic segmentation benchmark  that has 20 object categories. We follow the “comp6” evaluation protocol, which is also used in [4, 8, 10]. The training set of PASCAL VOC 2012 and the additional segmentation annotations from  are used for training and evaluation as in [4, 8, 10]. Two scenarios are studied: semantic segmentation and simultaneous detection and segmentation.
In the experiments of semantic segmentation, category labels are assigned to all the pixels in the image, and the accuracy is measured by region IoU scores .
We first study using the “ZF SPPnet” model  as our feature extractor. This model is based on Zeiler and Fergus’s fast model  but with the SPP layer . It has five convolutional layers and three fc layers. This model is released with the code of . We note that the results in R-CNN  and SDS  use the “AlexNet”  instead. To understand the impacts of the pre-trained models, we report their object detection mAP on the val set of PASCAL VOC 2012: SPP-Net (ZF) is 51.3%, R-CNN (AlexNet) is 51.0%, and SDS (AlexNet) is 51.9%. This means that both pre-trained models are comparable as generic feature extractors. So the following gains of CFM are not simply due to pre-trained models.
|no-CFM||CFM (A)||CFM (B)|
|ZF SPPnet||VGG net|
|ZF SPPnet||VGG net|
|conv time||fc time||total time|
|SDS (AlexNet) ||17.8s||0.14s||17.9s|
|CFM, (ZF, 5 scales)||0.29s||0.09s||0.38s|
|CFM, (ZF, 1 scale)||0.04s||0.09s||0.12s|
|CFM, (VGG, 5 scales)||1.74s||0.36s||2.10s|
|CFM, (VGG, 1 scale)||0.21s||0.36s||0.57s|
To show the effect of the CFM layer, we present a baseline with no CFM - in our Design B, we remove the CFM layer but still use the same entire pipeline. We term this baseline as the “no-CFM” version of our method. Actually, this baseline degrades to the original SPP-net usage , except that the definitions of positive/negative samples are for segmentation. Table 4 compares the results of no-CFM and the two designs of CFM. We find that the CFM has obvious advantages over the no-CFM baseline. This is as expected, because the no-CFM baseline has not any segment-based feature. Further, we find that the designs A and B perform just comparably, while A needs to compute two pathways of the fc layers. So in the rest of this paper, we adopt Design B for ZF SPPnet.
In Table 4 we evaluate our method using different region proposal algorithms. We adopt two proposal algorithms: Selective Search (SS) , and Multiscale Combinatorial Grouping (MCG) . Following the protocol in , the “fast” mode is used for SS, and the “accurate” mode is used for MCG. Table 4 shows that our method achieves higher accuracy on the MCG proposals. This indicates that our feature masking method can exploit the information generated by more accurate segmentation proposals.
In Table 4 we also evaluate the impact of pre-trained networks. We compare the ZF SPPnet with the public VGG-16 model 222www.robots.ox.ac.uk/~vgg/research/very_deep/. Recent advances in image classification have shown that very deep networks  can significantly improve the classification accuracy. The VGG-16 model has 13 convolutional and 3 fc layers. Because this model has no SPP layer, we consider its last pooling layer () as a special SPP layer which has a single-level pyramid of . In this case, our Design B does not apply because there is no coarser level. So we apply our Design A instead. Table 4 shows that our results improve substantially when using the VGG net. This indicates that our method benefits from the more representative features learned by deeper models.
In Table 4 we evaluate the impact of image scales. Instead of using the 5 scales, we simply extract features from single-scale images whose shorter side is . Table 4 shows that our single-scale variant has negligible degradation. But the single-scale variant has a faster computational speed as in Table 4.
|SDS (AlexNet + MCG) ||51.6||63.3||25.7||63.0||39.8||59.2||70.9||61.4||54.9||16.8||45.0||48.2||50.5||51.0||57.7||63.3||31.8||58.7||31.2||55.7||48.5|
|CFM (ZF + SS)||53.5||63.3||21.5||59.1||40.3||52.4||68.6||55.4||66.6||25.4||60.5||48.5||60.0||53.6||58.6||59.8||40.5||68.6||31.7||49.3||53.6|
|CFM (ZF + MCG)||55.4||65.2||23.5||59.0||40.4||61.1||68.9||57.9||70.8||23.9||59.4||44.7||66.2||57.5||62.1||57.6||44.1||64.5||42.5||52.9||55.7|
|CFM (VGG + MCG)||61.8||75.7||26.7||69.5||48.8||65.6||81.0||69.2||73.3||30.0||68.7||51.5||69.1||68.1||71.7||67.5||50.4||66.5||44.4||58.9||53.5|
Next we compare with the state-of-the-art results on the PASCAL VOC 2012 test set in Table 5. Here SDS  is the previous state-of-the-art method on this task, and OP  is a leading non-CNN-based method. Our method with ZF SPPnet and MCG achieves a score of 55.4. This is 3.8% higher than the SDS result reported in  which uses AlexNet and MCG. This demonstrates that our CFM method can produce effective features without masking raw-pixel images. With the VGG net, our method has a score of 61.8 on the test set.
Besides the high accuracy, our method is much faster than SDS. The running time of the feature extraction steps in SDS and our method is shown in Table 4
. Both approaches are run on an Nvidia GTX Titan GPU based on the Caffe library. The time is averaged over 100 random images from PASCAL VOC. Using 5 scales, our method with ZF SPPnet is faster than SDS; using 1 scale, our method with ZF SPPnet is faster than SDS and is more accurate. The speed gain is because our method only needs to compute the feature maps once. Table 4 also shows that our method is still feasible using the VGG net.
Concurrent with our work, a Fully Convolutional Network (FCN) method  is proposed for semantic segmentation. It has a score (62.2 on test set) comparable with our method, and has a fast speed as it also performs convolutions once on the entire image. But FCN is not able to generate instance-wise results, which is another metric evaluated in . Our method is also applicable in this case, as evaluated below.
In the evaluation protocol of simultaneous detection and segmentation , all the object instances and their segmentation masks are labeled. In contrast to semantic segmentation, this scenario further requires to identify different object instances in addition to labeling pixel-wise semantic categories. The accuracy is measured by mean APr score defined in .
We report the mean APr results on VOC 2012 validation set following , as the ground-truth labels for the test set are not available. As shown in Table 6, our method has a mean APr of 53.2 when using ZF SPPnet and MCG. This is better than the SDS result (49.7) reported in . With the VGG net, our mean APr is 60.7, which is the state-of-the-art result reported in this task. Note that the FCN method  is not applicable when evaluating the mean APr metric, because it cannot produce object instances.
The semantic categories in natural images can be roughly divided into objects and stuff. Objects have consistent shapes and each instance is countable, while stuff has consistent colors or textures and exhibits as arbitrary shapes, e.g., grass, sky, and water. So unlike an object, a stuff region is not appropriate to be represented as a rectangular region or a bounding box. While our method can generate segment features, each segment is still associated with a bounding box due to its way of generation. When the region/segment proposals are provided, it is rarely that the stuff can be fully covered by a single segment. Even if the stuff is covered by a single rectangular region, it is almost certain that there are many pixels in this region that do not belong to the stuff. So stuff segmentation has issues different from object segmentation.
Next we show a generalization of our framework to address this issue involving stuff. We can simultaneously handle objects and stuff by a single solution. Especially, the convolutional feature maps need only to be computed once. So there will be little extra cost if the algorithm is required to further handle stuff.
Our generalization is to modify the underlying probabilistic distributions of the samples during training. Instead of treating the samples equally, our training will bias toward the proposals that can cover the stuff as compact as possible (discussed below). A Segment Pursuit procedure is proposed to find the compact proposals.
We treat stuff as a combination of multiple segment proposals. We expect that each segment proposal can cover a stuff portion as much as possible, and a stuff can be fully covered by several segment proposals. At the same time, we hope the combination of these segment proposals is compact - the fewer the segments, the better.
We first define a candidate set of segment proposals (in a single image) for stuff segmentation. We define a “purity score” as the IoU ratio between a segment proposal and the stuff portion that is within the bounding box of this segment. Among all the segment proposals in a single image, those having high purity scores () with stuff consist of the candidate set for potential combinations.
To generate one compact combination from this candidate set, we adopt a procedure similar to the matching pursuit [23, 17]. We sequentially pick segments from the candidate set without replacement. At each step, the largest segment proposal is selected. This selected proposal then inhibits its highly overlapped proposals in the candidate set (they will not be selected afterward). In this paper, the inhibition overlap threshold is set as IoU=0.2. The process is repeated till the remaining segments all have areas smaller than a threshold, which is the average of the segment areas in the initial candidate set (of that image). We call this procedure segment pursuit.
Figure 4 (b) shows an example if segment proposals are randomly sampled from the candidate set. We see that there are many small segments. It is harmful to define these small, less discriminative segments as either positive or negative samples (e.g., by IoU) - if they are positive, they are just a very small part of the stuff; if they are negative, they share the same textures/colors as a larger portion of the stuff. So we prefer to ignore these samples in the training, so the classifier will not bias toward any side about these small samples. Figure 4 (c) shows the segment proposals selected by segment pursuit. We see that they can cover the stuff (grass here) by only a few but large segments. We expect the solver to rely more on such a compact combination of proposals.
However, the above process is deterministic and can only give a small set of samples from each image. For example, in Figure 4 (c) it only provides 5 segment proposals. In the fine-tuning process, we need a large number of stochastic
samples for the training. So we inject randomness into the above segment pursuit procedure. In each step, we randomly sample a segment proposal from the candidate set, rather than using the largest. The picking probability is proportional to the area size of a segment (so a larger one is still preferred). This can give us another compact combination in a stochastic way. Figure4 (d) shows an example of the segment proposals generated in a few trials.
All the segment proposals given by this way are considered as the positive samples of a category of stuff. The negative samples are the segment proposals whose purity scores are below 0.3. These samples can then be used for fine-tuning and SVM training as detailed below.
|mean on †||15.2||29.2||32.4||37.2||40.4||46.1||49.5|
During the fine-tuning stage, in each epoch each image generates a stochastic compact combination. All the segment proposals in this combination for all images consist of the samples of this epoch. These samples are randomly permuted and fed into the SGD solver. Although now the samples appear mutually independent to the SGD solver, they are actually sampled jointly by the rule of segment pursuit. Their underlying probabilistic distributions will impact the SGD solver. This process is repeated for each epoch. For the SGD solver, we halt the training process after 200k mini-batches. For SVM training, we only use the single combination given by the deterministic segment pursuit.
Using this way, we can treat object+stuff segmentation in the same framework as for object-only. The only difference is that the stuff samples are provided in a way given by segment pursuit, rather than purely randomly. To balance different categories, the portions of objects, stuff, and background samples in each mini-batch are set to be approximately 30%, 30%, and 40%. The testing stage is the same as in the object-only case. While the testing stage is unchanged, the classifiers learned are biased toward those compact proposals.
We conduct experiments on the newly labeled PASCAL-CONTEXT dataset  for joint object and stuff segmentation. In this enriched dataset, every pixel is labeled with a semantic category. It is a challenging dataset with various images, diverse semantic categories, and balanced ratios of object/stuff pixels. Following the protocol in , the semantic segmentation is performed on the most frequent 59 categories and one background category (Table 7). The segmentation accuracy is measured by mean IoU scores over the 60 categories. Following , the mean of the scores over a subset of 33 easier categories (identified by ) is reported in this 60-way segmentation task as well. The training and evaluation are performed on the train and val sets respectively. We compare with two leading methods - SuperParsing  and OP , whose results are reported in . For fair comparisons, the region refinement  is not used in all methods. The pasting scheme is the same as in OP . In this comparison, we ignore R-CNN  and SDS  because they have not been developed for stuff.
Table 7 shows the mean IoU scores. Here “no-CFM” is our baseline (no CFM, no segment pursuit); “CFM w/o SP” is our CFM method but without segment pursuit; and “CFM” is our CFM method with segment pursuit. When segment pursuit is not used, the positive stuff samples are uniformly sampled from the candidate set (in which the segments have purity scores 0.6).
SuperParsing  gets a mean score of 15.2 on the easier 33 categories, and the overall score is unavailable in . The OP method  results in 29.2 on the easier 33 categories and 18.1 overall, as reported in . Both methods are not based on CNN features.
For the CNN-based results, the no-CFM baseline (20.7, with ZF and SS) is already better than OP (18.1). This is mainly due to the generic features learned by deep networks. Our CFM method without segment pursuit improves the overall score to 24.0. This shows the effects of the masked convolutional features. With our segment pursuit, the CFM method further improves the overall score to 26.6. This justifies the impact of the samples generated by segment pursuit. When replacing the ZF SPPnet by the VGG net, and the SS proposals by MCG, our method yields an over score of 34.4. So our method benefits from deeper models and more accurate segment proposals. Some of our results are shown in Figure 5.
It is worth noticing that although only mean IoU scores are evaluated in this dataset, our method is also able to generate instance-wise results for objects.
Additional Results. We also run our trained model on an external dataset of MIT-Adobe FiveK , which consists of images taken by professional photographers to cover a broad range of scenes, subjects, and lighting conditions. Although our model is not trained for this dataset, it produces reasonably good results (see Figure 6).
We have presented convolutional feature masking, which exploits the shape information at a late stage in the network. We have further shown that convolutional feature masking is applicable for joint object and stuff segmentation.
We plan to further study improving object detection by convolutional feature masking. Exploiting the context information provided by joint object and stuff segmentation would also be interesting.