Fully Convolution Network (FCN) has evolved to be the de facto network architecture in semantic segmentation after it is successfully adopted in . As it is difficult (usually fails) to train a strong FCN from scratch, Long et al. simply adapts the architecture of pre-trained classification network (e.g. VGG-16  trained on ImageNet  etc.). Even though FCN significantly advances the performance of semantic segmentation, here we discuss its possible drawbacks so that we can build stronger segmentation network architectures.
First and foremost, the pre-trained CNN is trained with low-resolution images (e.g. 224 224 pixels), whereas input segmentation images are usually in high resolution (e.g. pixels). The simple adaptation techniques adopted in FCN cannot effectively address this domain gap, which leads to less optimized segmentation performance of FCN. To elaborate, the feature maps that are used for classification in FCN have limited contextual fields, so predictions are inconsistent for local ambiguous regions. To close such a domain gap, one option is to adapt (downsample) the inputs. Unfortunately, this practice is found to deteriorate the performance of small-size objects. A more plausible and popular option is to modify network architectures (of FCN) so that they are fit for high-resolution inputs, and we will review and discuss these works in Section 2.
Additionally, Long et al. 
adopts two extra skip connections in FCN-8s to aggregate three scale contextual predictions. Nonetheless, such limited scale fusion in segmentation networks is not expected to handle the significant variance of object scales across different images.
Next, we present our Improved FCN (IFCN) to address those issues. We take the adaptation of VGG-16  as an example to demonstrate the architecture of IFCN.
1.1 IFCN vs FCN
The key modification to FCN is that IFCN plugs in a context network between the pre-trained CNN and upsample layers. Thus, IFCN entails new parameters, which is crucial to fill the domain gap during the network fine-tuning on high-resolution segmentation images. The context network is a densely branched convolution network. Specifically, it is stacked with multiple convolution blocks, and shortcut branches are further added from each intermediate feature map. Functionally speaking, context network is able to significantly expand the receptive fields of feature maps, which is essential to contextualize local semantic predictions. In addition, shortcut branches are also important to ease the optimization difficulty of context network, as they provide shortcut paths for the propagation of error signals. Meanwhile, those shortcut branches enable IFCN to make predictions based on rich scale contexts. In consequence, IFCN is able to converge to a significantly better local optima than FCN on standard semantic segmentation images.
1.2 IFCN-xs vs FCN-xs
We further add more skip connections from early feature maps of pre-trained CNN so that richer scale contexts are incorporated in semantic predictions. Similar to the notation of FCN-xs, we symbolize our segmentation network as IFCN-xs, where x denotes the sliding stride of the finest-resolution feature maps involved in prediction. Let’s take FCN-8s and IFCN-8s for illustrative comparison. FCN-8s has two skip connections from feature mapspool3 and pool4. IFCN-8s, on the other hand, adds skip connections for all the following feature maps starting with pool3. In this perspective, IFCN-xs leverages significantly richer scale context for predictions than FCN-xs. As each feature map carries some distinct scale information, they are complementary to collectively handle the scale variance of objects. It is important to note that these skip connections incorporate very limited computation overhead. We will present the details of IFCN-xs in Section 3.2.
With these architecture modifications, IFCN marries context aggregation as well as rich-scale contextual prediction in an elegant framework. IFCN delivers a strong segmentation network architecture and it achieves new state-of-the-arts on standard semantic segmentation datasets including ADE20K (ImageNet Scene Parsing) , Pascal Context , VOC 2012  and SUN-RGBD . In Section 4
, we also present detailed study to the architecture design of IFCN. Overall, IFCN doesn’t involve any new computation layers comparing to FCN. Thus, it can be easily implemented with current deep learning libraries. We will release the code and model upon the acceptance of this paper.
|Receptive field (px)|
|pool3||conv4_1||conv4_2||conv4_3||pool4||conv5_1||conv5_2||conv5_3||pool5||fc6 & fc7|
2 Review of FCN and its developments
The pre-trained CNN is optimized for the classification purpose of low-resolution images (e.g. pixels), whereas input images for semantic segmentation are usually in high resolution. Originally in , Long et al. didn’t take this domain gap into consideration, so they simply adapted the architecture of pre-trained CNN in the following aspects: (1), adapt the fc layer to its equivalent conv layer (with size ); (2), alter the output channel of last fc layer; (3), add two skip connections from low-level feature maps. Then, the shared parameters (with pre-trained CNN) are transferred to initialize FCN. Finally, FCN is fine-tuned on target segmentation datasets. Such simple adaptation approach cannot afford to fill the domain gap, which mainly contributes to the less optimized performance of FCN. To elaborate, the feature maps are exposed to limited contexts, so their element-wise predictions are expected to suffer from local ambiguity. On top of pre-trained CNN, it’s essential to introduce an extra module to distill context into local features. Based on this principle, researchers have proposed different interesting solutions.
One representative branch of work is to leverage Fully Connected CRF  (FC-CRF) to contextualize the unary predictions of FCN. For example, Chen et al.  applied FC-CRF to the unary predictions of DeepLab network, and they observed obvious improved visual quality of label prediction maps on object segmentation datasets (e.g. Pascal VOC ). Subsequently, Zheng et al. 
formulated FC-CRF as a Recurrent Neural Network (CRF-RNN) so that it can be jointly learned with FCN. Even though FC-CRF is effective towards refining the label maps, they are more like a post-processing refinement step. Thus, these works are orthogonal to our work, which can be used to further improve the performance of IFCN.
One branch of work introduces new computational layers to achieve contextual modeling, and then they integrate it with FCN to yield an end-to-end trainable segmentation network. For example, Liu et al.  adopted local convolution layers to approximate the mean field algorithm for pairwise terms in deep parsing network (DPN). Lin et al.  inserted convolution layers to model the semantic compatibility between image regions. Visin et al.  and Shuai et al.  leveraged RNNs to propagate local context in feature maps. Recently, Yu et al.  utilized dilated convolution kernels to expand the receptive field of feature maps. All these computational layers are designed to encode extra context to local features, and they brought noticeable performance benefits to FCN architectures. Similar to them, the context network in IFCN achieves context aggregation as well. Different from , our context network has multiple shortcut connections, which allow it to have deeper architecture by mitigating its optimization difficulties. Thus, our context network is able to expand the receptive fields significantly larger than those networks. Moreover, shortcut connections enable IFCN to fuse rich-scale contextual predictions, whereas those network architectures don’t have such property. In contrast to , IFCN is more efficient as features (in feature maps) are processed in parallel rather than sequentially as in .
Another popular architecture modification is to combine multi-scale predictions. For example, Farabet et al.  and Lin et al.  adopted multi-resolution input (image pyramid) approach. A recent pixel-level architecture - PixelNet  followed the hypercolumn feature approach . Both approaches incur either significantly higher computation time or larger memory footprints, thus they simply combine few limited contextual predictions in practice. By contrast, IFCN adopts dense skip connections to fuse multi-scale predictions, which is a very economic as well as an effective approach to fuse very rich contextual predilections. As IFCN adopts linear fusion strategy like in , its performance can be further enhanced by adopting more advanced fusion method like in .
3.1 Context Networks
The pre-trained CNN is optimized for the classification purpose of low-resolution images, so its network architecture is less competitive on semantic segmentation task involving high-resolution images. Take the well-known FCN  for demonstration, the feature map that is used for predictions (fc7, c.f. Table 1) have a contextual view of pixels, which is not sufficiently large for every feature to make robust prediction. Besides, previous literatures  demonstrate that the contextual field of feature maps plays a critical role in the segmentation performance. Based on such principle, we attach a context network on top of the pre-trained CNN, whose key goal is to effectively enlarge the receptive fields of feature maps.
We adopt the basic conv block (Conv + BN + ReLU) to build our context network. The rationales behind such architecture choice are: (1), Conv can aggregate neighborhood context. In detail, suppose , where is Conv kernel (with size ), symbolizes Conv operation, and are 1-D input and output signals, and they have the same dimension. Mathematically, , thus it can be interpreted as a contextualized ; (2), Conv has been broadly and successfully devised and used in many literatures  to perform context aggregation; (3), Conv is more efficient compared to recursive / recurrent NN  to aggregate context.
Then, we are able to expand the contextual fields progressively by stacking multiple conv blocks. Unfortunately, the resulting segmentation network doesn’t work properly according to our experiments (c.f. Section 4.3), which also aligns with the discoveries by Long et al. . There could be two possible reasons that lead to such undesirable behavior: (1), the network training is hard due to gradient vanishing problem when the context network is deep; (2), the final feature map generated by the context network is expected to have wide (global) range of contextual views, which makes the features less discriminative for local predictions. To address these issues, we introduce shortcut branches emanating from intermediate feature maps. The architecture of context network is shown in Figure 1. Specifically, those shortcut branches are responsible for locally predicting feature maps. Then these predictions are summed to generate the fused prediction map. They can be mathematically represented as
where is the fused prediction map (context in Figure 1), is the quantity of conv blocks (i.e. depth of context network), represents the shortcut function, denotes the parameters of -th shortcut branch and is the output feature map of -th conv block. From the equation, we can see that is directly connected to the error signals (exclude upsample layers), thus its preceding conv block receives strong supervision signals. From this perspective, the gradient vanishing problem is greatly mitigated, so the context network can be effectively trained. Moreover, the fused prediction map combines multiple decisions from . Knowing that each preserves different scale context, their fused prediction map is expected to be more robust.
3.2 Dense skip connections
FCN-8s  adds two skip connections to FCN(-32s) so that low-level and mid-level feature maps are also utilized to make predictions. Specifically, FCN-8s fuses predictions from pool3, pool4 and fc7. As these feature maps encode diverse scale context (c.f. Table 1), it’s advantageous to combine their complementary decisions to account for objects with varied scales. In practice, FCN-8s significantly outperforms FCN(-32s).
Considering that the scales of objects vary dramatically across images and scenes, we are unable to utilize a few (e.g. 3 in FCN-8s) fixed scale contexts to handle such significant scale variances. Next, we investigate the receptive fields of intermediate feature maps in pre-trained VGG-16 (FCN). From Table 1, we observe that each feature map carries distinct scale information, and they are complementary to each other. Therefore, we add skip connections from these feature maps to prediction maps, so that rich-scale contextual prediction is collectively achieved. Such architecture modification gives rise to our IFCN-xs, a similar notation to FCN-xs, where ‘xs’ denotes that the finest resolution of prediction map has a sliding stride x. The architecture of IFCN-xs is illustrated in Figure 2. In comparison with FCN-xs, IFCN-xs entails more shortcut connections. These dense skip connections enable IFCN-xs to leverage very rich-scale context to make predictions, which is essential to tackle the high scale variance of objects. It’s also important to note that these dense skip connections involve minor extra computation overhead comparing to FCN-xs.
In the end, IFCN-xs is trained with pixel-wise loss (similar to FCN-xs). To distribute more attention for infrequent classes, we modulate the pixel-wise loss according to its rareness magnitude as in . This practice is economic and it is essential to significantly boost the recognition performance of rare classes. We follow the 85% -15% rule to determine the rare categories, and we refer the readers to  for detailed description.
4 Architecture Evaluation on ADE20K
ADE20K  is a recently created large-scale dataset for ImageNet scene parsing challenge. The dataset contains 20210 training, 2000 validation and 3352 test images. Each pixel is annotated with one of 150 semantic categories including thing classes as well as stuff classes. In order to modulate the rareness weighted loss , those classes whose frequency is less than are considered as rare. We report the results on validation images.
Experimental setups: The hidden dimension of conv
blocks in context network is fixed to 512. The new parameters engaged in the context network and skip connections are randomly initialized (Gaussian distribution with variance). We use convolution transpose (deconvolution) kernels  to perform upsampling operation. IFCN is trained with SGD with momentum (batch size 10). The learning rate is initialized to be
, and it is decreased by 10 times after 15 and 20 epoches (25 epoches in total). The momentum is fixed to 0.9. Meanwhile, higher learning rate () is used for newly-initialized parameters, i.e. context network and skip connections. In order to observe clear benefits of network architectures, we report the segmentation performance of their unary predictions
. In addition, IFCN is only trained with provided segmentation masks. Images are resized to have maximum length of 512 pixels, and they are zero padded to
pixels to allow for batch processing. We randomly flip the images horizontally (on the fly) to augment the training images. The statistics (mean and variance) in batch normalization (BN) layer is updated after the network is converged.
Evaluation: Three performance metrics are used to evaluate our IFCN-xs: overall pixel accuracy (Pixel Acc.), mean class accuracy (Mean Acc.) and Mean IOU. We refer the readers to  for mathematical definitions.
4.1 Architecture discussion of context network
We evaluate several key principles for designing a good context network (i.e. densely branched convolution network in this paper). For this purpose, we introduce the network architecture IFCN-8s-A, which has a similar structure to FCN-8s except that a context network is engaged in IFCN-8s-A. Thus, their performance gap clearly demonstrates how the introduced context network influences the segmentation behavior.
|Context network||Receptive Field (px)||Mean IOU|
Architecture Shapes. We build multiple typical context networks, which expand the same size of receptive fields whereas they have different network depths. A large receptive field (expanded by IFCN-8s-A) is carefully designed to make sure that nearly all elements in feature maps are accessible to full-image context. 111 Note that off-centering pixels require significantly larger receptive fields than centering pixels to have the same actual contextual views. In order to make sure that all features engage full-image context, the network is expected to expand a receptive field that is significantly () larger than the size of input images. Our preliminary experiments show that the segmentation performance of IFCN-8s-A degrades as their receptive fields go beyond this size. We demonstrate their settings as well as the segmentation performances of their corresponding IFCN-8s-A in Table 2. From which, we make the following observations.
First and foremost, all IFCN-8s-A has a significantly larger receptive field than that of FCN-8s. In consequence, they outperform FCN-8s by a noticeable margin, which clearly shows the significance of context networks (receptive field expansion) inside the architecture design of segmentation networks. In addition, deep context networks performs better than shallow alternatives when they expand equally large size of receptive fields. This result can be interpreted from two aspects. First, the shallower network has fewer skip connections, and it generates fewer feature maps. Thus, some informative scale contextual predictions are not fused into final decisions. Meanwhile, large conv block (in shallow networks) refers to higher-order context aggregation, and it involves more parameters. These factors are expected to add difficulty to the training of context networks.
Batch Normalization ( BN )  is essential to accelerate training as well as to improve the segmentation performance. As depicted in Figure 1, we plug in a BN layer right after each convolution layer in the conv block. In experiments, we find that the context network with BN performs significantly better than that without engaging BN.
4.2 Does performance improvement simply originate from engaging more parameters?
|Context network||Tied||#Params||Mean IOU.|
First, it’s crucial to introduce new parameters, which are responsible for filling the domain gap during the fine-tuning of segmentation networks (IFCN). To illustrate this situation, let’s directly compare FCN-8s and IFCN-8s-A. In FCN-8s, the pre-trained fc layers (fc6 and fc7) involves significantly more parameters than any context networks listed in Table 2. IFCN-8s-A 222It’s important to note that we have removed the pre-trained fc layers (both fc6 and fc7) in IFCN (IFCN-8s-A). (plugged in with these context networks), on the other hand, outperforms FCN-8s by a large margin.
Besides, we believe that the network architecture outweighs the magnitude of parameters in terms of boosting the performance. To validate this claim, we conduct two controlled experiments. To be specific, we adopt the architecture of to be the basic component of context networks, where denotes the quantity of conv blocks (i.e. network depth). We further plug them into IFCN-8s-A, and compare their segmentation performance in Table 3. We interpret the results from two perspectives.
Tied Parameters. First, we discuss the context network with tied parameters. Under this scenario, the amount of parameters engaged by context networks is significantly reduced. In our experiment, the tied network has the same amount of parameters with . However, the former network largely outperforms the alternative, as the tied network expand significantly larger receptive fields than . This result convinces us to believe that the huge performance boost mainly results from the architecture of context networks.
Wide context network. Then, we train a wide network whose architecture configuration is . Different from previous settings, the hidden dimension of wide network is increased from to . Under this scenario, the wide network involves magnitude larger parameters. However, it fails to generate significantly better result. Therefore, it’s fair to conclude that the increase of network parameters is not the key contributor towards performance boost.
4.3 IFCN-xs: Is dense skip connection necessary ?
|Network||Pixel Acc.||Mean Acc.||Mean IOU|
To answer this question, we investigate three parts of shortcut connections in IFCN-xs.
IFCN-8s-A vs IFCN-8s-B. We first build IFCN-8s-B by removing the shortcut branches in IFCN-8s-A. Then we directly compare their segmentation performance. As shown in Table 4, the performance of IFCN-8s-B significantly lags behind that of IFCN-8s-A. Such a performance gap clearly demonstrates that the shortcut branches indeed improves the training of context networks. Besides, it also indicates that fusion of multiple scale contextual predictions is beneficial.
IFCN-8s vs IFCN-8s-A. Next, we compare these two network architectures. The architecture of FCN-8s is presented in Figure 2. IFCN-8s-A follows the architecture design of FCN-8s . In Table 4, we observe a noticeable performance improvement when we switch IFCN-8s-A to IFCN-8s. This result shows that the dense shortcut connections enable IFCN to fuse rich-scale contextual predictions, which is crucial towards achieving high performance.
IFCN-xs. Finally, we empirically discuss which skip connection should be included in the IFCN-xs architecture. Considering that it’s extremely expensive as well as unnecessary to evaluate every skip connection, we simply evaluate the skip connections in one block. In more details, the feature maps are considered to be in the same block as long as they have the same spatial resolution. Take VGG-16 in Table 1 as an example, feature maps pool4, conv5_1, conv5_2, conv5_3 have the same resolution, so skip connections emanating from them are deemed as in the same block. By adding these skip connections progressively, we are able to produce a number of IFCN-xs whose architecture is illustrated in Figure 2. Among all the architectures, IFCN-8s performs the best. Though IFCN-4s entails more skip connections than IFCN-8s, it however achieves inferior segmentation results. We believe that the feature maps attached by the added skip connections (in IFCN-4s, namely pool2, conv3_1, conv3_2, con3_3) incorporate too limited context (less than pixels, c.f. Table 1), thus the predictions based on them is expected to be noisy. It’s empirically not advisable to include these skip connections in the IFCN architecture.
4.4 Can the proposed architecture work for deeper pre-trained network?
Due to the recent success of residue learning , this question arises naturally as we are easily accessible to very deep pre-trained CNN nowadays. For example, the pre-trained deep networks (e.g. ResNet-101) have already expanded very large receptive fields. In this scenario, it’s curious and important to know whether the proposed architecture design (IFCN) is still able to bring performance benefits, if not as significant as that in VGG-16.
We adapt the pre-trained ResNets (ResNet-50  and ResNet-101 ) to two different architectures: FCN-8s and IFCN-8s. As shown in Table 5: IFCN-8s consistently and significantly outperforms FCN-8s, which again validates the solid contributions of this paper. This result is inspiring and interesting considering that FCN-8s adapted from deep ResNet is already strong. Knowing that the pre-trained ResNets are trained from low-resolution images, the introduced context network is essential and effective to adapt the feature maps so that they are optimized for the segmentation purpose of high-resolution images. Meanwhile, the fusion of rich-scale contextual predictions achieved by dense skip connections also contribute significantly to the performance boost.
5 Results on segmentation benchmarks
We evaluate our IFCN-8s on standard semantic segmentation datasets, and the architecture of its context network is . The same experimental setups (as in controlled experiments, c.f. Section 4) are used to train IFCN-8s over different datasets. It’s worth mentioning that wider context network has higher capacity, and it can further generate slightly better performance (c.f. Table 3).
|Methods||Pixel Acc.||Mean Acc.||Mean IoU|
ADE20K results are listed in Table 6.
|Methods||Pixel Acc.||Mean Acc.||Mean IoU|
|DeepLab + CRF ||n/a||n/a||39.6|
|UoA-Context + CRF ||71.5||53.9||43.3|
Pascal Context results are shown in Table 7. Pascal Context  has 10103 images, out of which 4998 images are used for training. The images are from Pascal VOC 2010 datasets, and they are re-labeled as pixel-wise segmentation maps which include 540 semantic classes (including the original 20 classes). Similar to Mottaghi et al. , we only consider the most frequent 59 classes in the dataset for evaluation. Based on the rareness identification rule in , those classes whose frequencies are lower than are identified as rare.
|Methods||Pixel Acc.||Mean Acc.||Mean IOU|
SUN-RGBD Results are reported in Table 8. SUN-RGBD  contains images from NYU depth V2 , Berkeley B3DO , SUN3D  as well as the newly captured images. It has 10335 images in total, out of which 5285 images are used for training. The rareness frequency threshold is fixed to based on the 85%-15% rule . We follow previous literatures  to consider 37 classes for evaluation. Note that we only use RGB modality as input.
|IFCN-8s + CRF||75.3|
|DeepLab-v1 + CRF ||71.6|
|UoA-Context + CRF||75.3|
|LRR + CRF ||75.9|
|IFCN-8s + CRF||80.0|
|LRR + CRF ||79.3|
|DeepLab-v2 + CRF ||79.7|
Pascal VOC 2012 results are presented in Table 9. Pascal VOC 2012 originally contains 2913 train and validation images, 1456 testing images. Each pixel belongs to one of the pre-defined object or background categories. We follow  to augment the training set from . Thus, we end up with having 12031 training images. The rareness frequency threshold of is set to . We submit our prediction maps to the evaluation servers to get the reported testing results. Detailed class-wise IOU (Jaccard) score can be retrieved in supplementary materials 333They can be downloaded from the author’s homepage..
5.1 Result Summary
IFCN vs FCN. IFCN-8s demonstrates significantly ( mIOU) better quantitative results than FCN-8s on all testing segmentation benchmarks. It’s important to note that these datasets cover wide range of scenarios that include object segmentation , outdoor scene parsing  as well as indoor scene labeling . Thus, it’s fair to conclude that IFCN is a versatilely better segmentation network architecture than FCN.
Next, we qualitatively compare their segmentation maps in Figure 3. As shown, FCN-8s struggles to make robust predictions for zoom-in objects (e.g. 1 and 4 example ), as it has limited receptive fields. IFCN-8s can reliably correct those errors. In addition, IFCN-8s is able to produce very detailed segmentation maps that have sharper boundaries than FCN-8s (e.g. 2 and 3 examples). We speculate that the dense skip connections originating from early layers of pre-trained CNNs play a significant role to achieve such appealing property of IFCN.
We further compare the inference time of IFCN-8s with FCN-8s in Table 10. As shown, IFCN-8s (VGG-16) takes reasonably longer time than FCN-8s to generate the prediction maps. We believe that such time consumption is competitive when it is positioned with other recent segmentation network architectures, e.g. DeepLab , DilatedNet  etc. The reported time statistics are based on the average feed-forward time for images in Pascal Context dataset  under a single Titan X (exclude the time consumption for image pre-processing).
IFCN vs State-of-the-arts. IFCN has also produced significantly better performance over other recently developed segmentation network architectures including DilatedNet , ParseNet , DeepLab , PixelNet , SegNet , etc. In the setting of VGG-16, the unary predictions of IFCN-8s already outperforms state-of-the-arts by a large margin on ADE20K, Pascal Context and SUN RGB-D datasets. On Pascal VOC 2012, IFCN-8s (+ CRF ) also ranks highly competitive among all of top methods. In the advanced setting where pre-trained CNN is initialized with ResNet-101 , IFCN further significantly advances state-of-the arts on all datasets. It’s important to note that IFCN has not been pre-trained with auxiliary segmentation data (e.g. Microsoft COCO dataset ) in all cases.
|Network||Inference Time (ms)||Mean IOU (%)|
In this paper, we discuss the possible limitations of de facto segmentation network - FCN. To address these architecture flaws, we present our Improved FCN (IFCN), which is a strong segmentation network architecture that marries effective context aggregation as well as rich-scale contextual predictions in an elegant framework. In detail, a context network is attached on top of pre-trained CNN to expand the receptive fields of feature maps. In addition, dense shortcut connections are added to enable IFCN to fuse very rich contextual predictions. Empirically, IFCN consistently and significantly outperforms FCN on standard semantic segmentation datasets including ADE20K, Pascal Context, Pascal VOC2012 and SUN-RGBD. Moreover, IFCN achieves new state-of-the-arts on all these datasets.
-  V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561, 2015.
-  A. Bansal, X. Chen, B. Russell, A. Gupta, and D. Ramanan. Pixelnet: Towards a general pixel-level architecture. arXiv preprint arXiv:1609.06694, 2016.
-  L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015.
-  L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915, 2016.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman.
The pascal visual object classes (voc) challenge.
International journal of computer vision, 88(2):303–338, 2010.
-  C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. IEEE transactions on pattern analysis and machine intelligence, 35(8):1915–1929, 2013.
-  G. Ghiasi and C. C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. In European Conference on Computer Vision, pages 519–534. Springer, 2016.
-  B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision, pages 297–312. Springer, 2014.
B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik.
Hypercolumns for object segmentation and fine-grained localization.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 447–456, 2015.
-  K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
-  S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
-  A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz, K. Saenko, and T. Darrell. A category-level 3d object dataset: Putting the kinect to work. In Consumer Depth Cameras for Computer Vision, pages 141–165. Springer, 2013.
-  P. Krähenbühl and V. Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS, 2011.
-  G. Lin, C. Shen, A. van dan Hengel, and I. Reid. Efficient piecewise training of deep structured models for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 2016.
-  T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014, pages 740–755. Springer, 2014.
-  W. Liu, A. Rabinovich, and A. C. Berg. Parsenet: Looking wider to see better. arXiv preprint arXiv:1506.04579, 2015.
-  Z. Liu, X. Li, P. Luo, C.-C. Loy, and X. Tang. Semantic image segmentation via deep parsing network. In Proceedings of the IEEE International Conference on Computer Vision, pages 1377–1385, 2015.
-  J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431–3440, 2015.
-  M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich. Feedforward semantic segmentation with zoom-out features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3376–3385, 2015.
-  R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
-  H. Noh, S. Hong, and B. Han. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1520–1528, 2015.
-  O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
-  A. Sharma, O. Tuzel, and M.-Y. Liu. Recursive context propagation network for semantic scene labeling. In Advances in Neural Information Processing Systems, pages 2447–2455, 2014.
-  B. Shuai, Z. Zuo, G. Wang, and B. Wang. Dag-recurrent neural networks for scene labeling. In Proceedings of the IEEE International Conference on Computer Vision, 2016.
-  N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, pages 746–760. Springer, 2012.
-  K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
S. Song, S. P. Lichtenberg, and J. Xiao.
Sun rgb-d: A rgb-d scene understanding benchmark suite.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 567–576, 2015.
-  F. Visin, M. Ciccone, A. Romero, K. Kastner, K. Cho, Y. Bengio, M. Matteucci, and A. Courville. Reseg: A recurrent neural network-based model for semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2016.
-  Z. Wu, C. Shen, and A. v. d. Hengel. Bridging category-level and instance-level semantic image segmentation. arXiv preprint arXiv:1605.06885, 2016.
-  J. Xiao, A. Owens, and A. Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of the IEEE International Conference on Computer Vision, pages 1625–1632, 2013.
-  S. Xie, X. Huang, and Z. Tu. Convolutional pseudo-prior for structured labeling. arXiv preprint arXiv:1511.07409, 2015.
-  J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328, 2014.
-  F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
-  S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1529–1537, 2015.
-  B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv:1608.05442, 2016.