Fully Convolutional Networks for Semantic Segmentation by Jonathan Long*, Evan Shelhamer*, and Trevor Darrell. CVPR 2015 and PAMI 2016.
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, improve on the previous best result in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves improved segmentation of PASCAL VOC (30 Flow, and PASCAL-Context, while inference takes one tenth of a second for a typical image.READ FULL TEXT VIEW PDF
We present a simple and effective framework for simultaneous semantic
In this work I propose a new way of using fully convolutional networks f...
Semantic segmentation of functional magnetic resonance imaging (fMRI) ma...
Increased information sharing through short and long-range skip connecti...
Global warming is rapidly reducing glaciers and ice sheets across the wo...
In this paper, we use fully convolutional neural networks for the semant...
We present a neighborhood similarity layer (NSL) which induces appearanc...
Fully Convolutional Networks for Semantic Segmentation by Jonathan Long*, Evan Shelhamer*, and Trevor Darrell. CVPR 2015 and PAMI 2016.
Implementation FCN via chainer
This project was developed for identifying vehicles in a video stream. The project is a corner stone for a real time vehicle tracking algorithm that employ semantic pixel-wise methods. This project solves the tracking problem for the Udacity final project in a different way that the general approach presented in the course. Instead of using the HOG features and other features extracted from the color space of the images, we used the U-Net which is a convolutional network for biomedical image
Convolutional networks are driving advances in recognition. Convnets are not only improving for whole-image classification [1, 2, 3], but also making progress on local tasks with structured output. These include advances in bounding box object detection [4, 5, 6], part and keypoint prediction [7, 8], and local correspondence [8, 9].
The natural next step in the progression from coarse to fine inference is to make a prediction at every pixel. Prior approaches have used convnets for semantic segmentation [10, 11, 12, 13, 14, 15, 16], in which each pixel is labeled with the class of its enclosing object or region, but with shortcomings that this work addresses.
We show that fully convolutional networks (FCNs) trained end-to-end, pixels-to-pixels on semantic segmentation exceed the previous best results without further machinery. To our knowledge, this is the first work to train FCNs end-to-end (1) for pixelwise prediction and (2) from supervised pre-training. Fully convolutional versions of existing networks predict dense outputs from arbitrary-sized inputs. Both learning and inference are performed whole-image-at-a-time by dense feedforward computation and backpropagation. In-network upsampling layers enable pixelwise prediction and learning in nets with subsampling.
This method is efficient, both asymptotically and absolutely, and precludes the need for the complications in other works. Patchwise training is common [10, 11, 12, 13, 16], but lacks the efficiency of fully convolutional training. Our approach does not make use of pre- and post-processing complications, including superpixels [12, 14], proposals [14, 15]
, or post-hoc refinement by random fields or local classifiers[12, 14]. Our model transfers recent success in classification [1, 2, 3] to dense prediction by reinterpreting classification nets as fully convolutional and fine-tuning from their learned representations. In contrast, previous works have applied small convnets without supervised pre-training [12, 13, 10].
Semantic segmentation faces an inherent tension between semantics and location: global information resolves what while local information resolves where. What can be done to navigate this spectrum from location to semantics? How can local decisions respect global structure? It is not immediately clear that deep networks for image classification yield representations sufficient for accurate, pixelwise recognition.
In the conference version of this paper , we cast pre-trained networks into fully convolutional form, and augment them with a skip architecture that takes advantage of the full feature spectrum. The skip architecture fuses the feature hierarchy to combine deep, coarse, semantic information and shallow, fine, appearance information (see Section 4.3 and Figure 3
). In this light, deep feature hierarchies encode location and semantics in a nonlinear local-to-global pyramid.
This journal paper extends our earlier work  through further tuning, analysis, and more results. Alternative choices, ablations, and implementation details better cover the space of FCNs. Tuning optimization leads to more accurate networks and a means to learn skip architectures all-at-once instead of in stages. Experiments that mask foreground and background investigate the role of context and shape. Results on the object and scene labeling of PASCAL-Context reinforce merging object segmentation and scene parsing as unified pixelwise prediction.
In the next section, we review related work on deep classification nets, FCNs, recent approaches to semantic segmentation using convnets, and extensions to FCNs. The following sections explain FCN design, introduce our architecture with in-network upsampling and skip layers, and describe our experimental framework. Next, we demonstrate improved accuracy on PASCAL VOC 2011-2, NYUDv2, SIFT Flow, and PASCAL-Context. Finally, we analyze design choices, examine what cues can be learned by an FCN, and calculate recognition bounds for semantic segmentation.
Our approach draws on recent successes of deep nets for image classification [1, 2, 3] and transfer learning [18, 19]. Transfer was first demonstrated on various visual recognition tasks [18, 19], then on detection, and on both instance and semantic segmentation in hybrid proposal-classifier models [5, 14, 15]. We now re-architect and fine-tune classification nets to direct, dense prediction of semantic segmentation. We chart the space of FCNs and relate prior models both historical and recent.
Fully convolutional networks To our knowledge, the idea of extending a convnet to arbitrary-sized inputs first appeared in Matan et al. , which extended the classic LeNet  to recognize strings of digits. Because their net was limited to one-dimensional input strings, Matan et al. used Viterbi decoding to obtain their outputs. Wolf and Platt  expand convnet outputs to 2-dimensional maps of detection scores for the four corners of postal address blocks. Both of these historical works do inference and learning fully convolutionally for detection. Ning et al.  define a convnet for coarse multiclass segmentation of C. elegans tissues with fully convolutional inference.
Fully convolutional computation has also been exploited in the present era of many-layered nets. Sliding window detection by Sermanet et al. , semantic segmentation by Pinheiro and Collobert , and image restoration by Eigen et al.  do fully convolutional inference. Fully convolutional training is rare, but used effectively by Tompson et al. 
to learn an end-to-end part detector and spatial model for pose estimation, although they do not exposit on or analyze this method.
Dense prediction with convnets Several recent works have applied convnets to dense prediction problems, including semantic segmentation by Ning et al. , Farabet et al. , and Pinheiro and Collobert ; boundary prediction for electron microscopy by Ciresan et al.  and for natural images by a hybrid convnet/nearest neighbor model by Ganin and Lempitsky ; and image restoration and depth estimation by Eigen et al. [23, 25]. Common elements of these approaches include
small models restricting capacity and receptive fields;
whereas our method does without this machinery. However, we do study patchwise training (Section 3.4) and “shift-and-stitch” dense output (Section 3.2) from the perspective of FCNs. We also discuss in-network upsampling (Section 3.3), of which the fully connected prediction by Eigen et al.  is a special case.
Unlike these existing methods, we adapt and extend deep classification architectures, using image classification as supervised pre-training, and fine-tune fully convolutionally to learn simply and efficiently from whole image inputs and whole image ground thruths.
Hariharan et al.  and Gupta et al.  likewise adapt deep classification nets to semantic segmentation, but do so in hybrid proposal-classifier models. These approaches fine-tune an R-CNN system  by sampling bounding boxes and/or region proposals for detection, semantic segmentation, and instance segmentation. Neither method is learned end-to-end. They achieve the previous best segmentation results on PASCAL VOC and NYUDv2 respectively, so we directly compare our standalone, end-to-end FCN to their semantic segmentation results in Section 5.
Combining feature hierarchies We fuse features across layers to define a nonlinear local-to-global representation that we tune end-to-end. The Laplacian pyramid  is a classic multi-scale representation made of fixed smoothing and differencing. The jet of Koenderink and van Doorn  is a rich, local feature defined by compositions of partial derivatives. In the context of deep networks, Sermanet et al.  fuse intermediate layers but discard resolution in doing so. In contemporary work Hariharan et al.  and Mostajabi et al.  also fuse multiple layers but do not learn end-to-end and rely on fixed bottom-up grouping.
FCN extensions Following the conference version of this paper , FCNs have been extended to new tasks and data. Tasks include region proposals , contour detection , depth regression , optical flow , and weakly-supervised semantic segmentation [35, 36, 37, 38].
In addition, new works have improved the FCNs presented here to further advance the state-of-the-art in semantic segmentation. The DeepLab models  raise output resolution by dilated convolution and dense CRF inference. The joint CRFasRNN  model is an end-to-end integration of the CRF for further improvement. ParseNet  normalizes features for fusion and captures context with global pooling. The “deconvolutional network” approach of  restores resolution by proposals, stacks of learned deconvolution, and unpooling. U-Net  combines skip layers and learned deconvolution for pixel labeling of microscopy images. The dilation architecture of  makes thorough use of dilated convolution for pixel-precise output without a random field or skip layers.
Each layer output in a convnet is a three-dimensional array of size , where and are spatial dimensions, and is the feature or channel dimension. The first layer is the image, with pixel size , and channels. Locations in higher layers correspond to the locations in the image they are path-connected to, which are called their receptive fields.
Convnets are inherently translation invariant. Their basic components (convolution, pooling, and activation functions) operate on local input regions, and depend only onrelative spatial coordinates. Writing
for the data vector at locationin a particular layer, and for the following layer, these functions compute outputs by
where is called the kernel size,
is the stride or subsampling factor, and
determines the layer type: a matrix multiplication for convolution or average pooling, a spatial max for max pooling, or an elementwise nonlinearity for an activation function, and so on for other types of layers.
This functional form is maintained under composition, with kernel size and stride obeying the transformation rule
While a general net computes a general nonlinear function, a net with only layers of this form computes a nonlinear filter, which we call a deep filter or fully convolutional network. An FCN naturally operates on an input of any size, and produces an output of corresponding (possibly resampled) spatial dimensions.
A real-valued loss function composed with an FCN defines a task. If the loss function is a sum over the spatial dimensions of the final layer,
, its parameter gradient will be a sum over the parameter gradients of each of its spatial components. Thus stochastic gradient descent oncomputed on whole images will be the same as stochastic gradient descent on , taking all of the final layer receptive fields as a minibatch.
When these receptive fields overlap significantly, both feedforward computation and backpropagation are much more efficient when computed layer-by-layer over an entire image instead of independently patch-by-patch.
We next explain how to convert classification nets into fully convolutional nets that produce coarse output maps. For pixelwise prediction, we need to connect these coarse outputs back to the pixels. Section 3.2 describes a trick used for this purpose (e.g., by “fast scanning” ). We explain this trick in terms of network modification. As an efficient, effective alternative, we upsample in Section 3.3, reusing our implementation of convolution. In Section 3.4 we consider training by patchwise sampling, and give evidence in Section 4.4 that our whole image training is faster and equally effective.
Typical recognition nets, including LeNet , AlexNet , and its deeper successors [2, 3], ostensibly take fixed-sized inputs and produce non-spatial outputs. The fully connected layers of these nets have fixed dimensions and throw away spatial coordinates. However, fully connected layers can also be viewed as convolutions with kernels that cover their entire input regions. Doing so casts these nets into fully convolutional networks that take input of any size and make spatial output maps. This transformation is illustrated in Figure 2.
Furthermore, while the resulting maps are equivalent to the evaluation of the original net on particular input patches, the computation is highly amortized over the overlapping regions of those patches. For example, while AlexNet takes ms (on a typical GPU) to infer the classification scores of a image, the fully convolutional net takes ms to produce a grid of outputs from a image, which is more than times faster than the naïve approach111Assuming efficient batching of single image inputs. The classification scores for a single image by itself take 5.4 ms to produce, which is nearly times slower than the fully convolutional version..
The spatial output maps of these convolutionalized models make them a natural choice for dense problems like semantic segmentation. With ground truth available at every output cell, both the forward and backward passes are straightforward, and both take advantage of the inherent computational efficiency (and aggressive optimization) of convolution. The corresponding backward times for the AlexNet example are ms for a single image and ms for a fully convolutional output map, resulting in a speedup similar to that of the forward pass.
While our reinterpretation of classification nets as fully convolutional yields output maps for inputs of any size, the output dimensions are typically reduced by subsampling. The classification nets subsample to keep filters small and computational requirements reasonable. This coarsens the output of a fully convolutional version of these nets, reducing it from the size of the input by a factor equal to the pixel stride of the receptive fields of the output units.
Dense predictions can be obtained from coarse outputs by stitching together outputs from shifted versions of the input. If the output is downsampled by a factor of , shift the input pixels to the right and pixels down, once for every such that . Process each of these inputs, and interlace the outputs so that the predictions correspond to the pixels at the centers of their receptive fields.
Although this transformation naïvely increases the cost by a factor of , there is a well-known trick for efficiently producing identical results [45, 4]. (This trick is also used in the algorithme à trous [46, 47] for wavelet transforms and related to the Noble identities  from signal processing.)
Consider a layer (convolution or pooling) with input stride , and a subsequent convolution layer with filter weights (eliding the irrelevant feature dimensions). Setting the earlier layer’s input stride to one upsamples its output by a factor of . However, convolving the original filter with the upsampled output does not produce the same result as shift-and-stitch, because the original filter only sees a reduced portion of its (now upsampled) input. To produce the same result, dilate (or “rarefy”) the filter by forming
(with and zero-based). Reproducing the full net output of shift-and-stitch involves repeating this filter enlargement layer-by-layer until all subsampling is removed. (In practice, this can be done efficiently by processing subsampled versions of the upsampled input.)
Simply decreasing subsampling within a net is a tradeoff: the filters see finer information, but have smaller receptive fields and take longer to compute. This dilation trick is another kind of tradeoff: the output is denser without decreasing the receptive field sizes of the filters, but the filters are prohibited from accessing information at a finer scale than their original design.
Although we have done preliminary experiments with dilation, we do not use it in our model. We find learning through upsampling, as described in the next section, to be effective and efficient, especially when combined with the skip layer fusion described later on. For further detail regarding dilation, refer to the dilated FCN of .
Another way to connect coarse outputs to dense pixels is interpolation. For instance, simple bilinear interpolation computes each output from the nearest four inputs by a linear map that depends only on the relative positions of the input and output cells:
where is the upsampling factor, and denotes the fractional part.
In a sense, upsampling with factor is convolution with a fractional input stride of . So long as is integral, it’s natural to implement upsampling through “backward convolution” by reversing the forward and backward passes of more typical input-strided convolution. Thus upsampling is performed in-network for end-to-end learning by backpropagation from the pixelwise loss.
Per their use in deconvolution networks (esp. ), these (convolution) layers are sometimes referred to as deconvolution layers. Note that the convolution filter in such a layer need not be fixed (e.g., to bilinear upsampling), but can be learned. A stack of deconvolution layers and activation functions can even learn a nonlinear upsampling.
In our experiments, we find that in-network upsampling is fast and effective for learning dense prediction.
In stochastic optimization, gradient computation is driven by the training distribution. Both patchwise training and fully convolutional training can be made to produce any distribution of the inputs, although their relative computational efficiency depends on overlap and minibatch size. Whole image fully convolutional training is identical to patchwise training where each batch consists of all the receptive fields of the output units for an image (or collection of images). While this is more efficient than uniform sampling of patches, it reduces the number of possible batches. However, random sampling of patches within an image may be easily recovered. Restricting the loss to a randomly sampled subset of its spatial terms (or, equivalently applying a DropConnect mask  between the output and the loss) excludes patches from the gradient.
If the kept patches still have significant overlap, fully convolutional computation will still speed up training. If gradients are accumulated over multiple backward passes, batches can include patches from several images. If inputs are shifted by values up to the output stride, random selection of all possible patches is possible even though the output units lie on a fixed, strided grid.
Sampling in patchwise training can correct class imbalance [10, 12, 11] and mitigate the spatial correlation of dense patches [13, 14]. In fully convolutional training, class balance can also be achieved by weighting the loss, and loss sampling can be used to address spatial correlation.
We explore training with sampling in Section 4.4, and do not find that it yields faster or better convergence for dense prediction. Whole image training is effective and efficient.
We cast ILSVRC classifiers into FCNs and augment them for dense prediction with in-network upsampling and a pixelwise loss. We train for segmentation by fine-tuning. Next, we add skips between layers to fuse coarse, semantic and local, appearance information. This skip architecture is learned end-to-end to refine the semantics and spatial precision of the output.
For this investigation, we train and validate on the PASCAL VOC 2011 segmentation challenge . We train with a per-pixel softmax loss and validate with the standard metric of mean pixel intersection over union, with the mean taken over all classes, including background. The training ignores pixels that are masked out (as ambiguous or difficult) in the ground truth.
|forward time||16 ms||100 ms||20 ms|
We begin by convolutionalizing proven classification architectures as in Section 3.
We consider the AlexNet222Using the publicly available CaffeNet reference model. architecture  that won ILSVRC12, as well as the VGG nets  and the GoogLeNet333We use our own reimplementation of GoogLeNet.
Ours is trained with less extensive data augmentation, and gets 68.5% top-1 and 88.4% top-5 ILSVRC accuracy.  which did exceptionally well in ILSVRC14.
We pick the VGG 16-layer net444 Using the publicly available version from the Caffe model zoo.
Using the publicly available version from the Caffe model zoo., which we found to be equivalent to the 19-layer net on this task. For GoogLeNet, we use only the final loss layer, and improve performance by discarding the final average pooling layer. We decapitate each net by discarding the final classifier layer, and convert all fully connected layers to convolutions. We append a convolution with channel dimension to predict scores for each of the PASCAL classes (including background) at each of the coarse output locations, followed by a (backward) convolution layer to bilinearly upsample the coarse outputs to pixelwise outputs as described in Section 3.3. Table I
compares the preliminary validation results along with the basic characteristics of each net. We report the best results achieved after convergence at a fixed learning rate (at least 175 epochs).
Our training for this comparison follows the practices for classification networks. We train by SGD with momentum. Gradients are accumulated over 20 images. We set fixed learning rates of , , and for FCN-AlexNet, FCN-VGG16, and FCN-GoogLeNet, respectively, chosen by line search. We use momentum , weight decay of or , and doubled learning rate for biases. We zero-initialize the class scoring layer, as random initialization yielded neither better performance nor faster convergence. Dropout is included where used in the original classifier nets (however, training without it made little to no difference).
Fine-tuning from classification to segmentation gives reasonable predictions from each net. Even the worst model achieved of the previous best performance. FCN-VGG16 already appears to be better than previous methods at 56.0 mean IU on val, compared to 52.6 on test . Although VGG and GoogLeNet are similarly accurate as classifiers, our FCN-GoogLeNet did not match FCN-VGG16. We select FCN-VGG16 as our base network.
The image-to-image learning setting includes high effective batch size and correlated inputs. This optimization requires some attention to properly tune FCNs.
We begin with the loss. We do not normalize the loss, so that every pixel has the same weight regardless of the batch and image dimensions. Thus we use a small learning rate since the loss is summed spatially over all pixels.
We consider two regimes for batch size. In the first, gradients are accumulated over 20 images. Accumulation reduces the memory required and respects the different dimensions of each input by reshaping the network. We picked this batch size empirically to result in reasonable convergence. Learning in this way is similar to standard classification training: each minibatch contains several images and has a varied distribution of class labels. The nets compared in Table I are optimized in this fashion.
However, batching is not the only way to do image-wise learning. In the second regime, batch size one is used for online learning. Properly tuned, online learning achieves higher accuracy and faster convergence in both number of iterations and wall clock time. Additionally, we try a higher momentum of , which increases the weight on recent gradients in a similar way to batching. See Table II for the comparison of accumulation, online, and high momentum or “heavy” learning (discussed further in Section 6.2).
We define a new fully convolutional net for segmentation that combines layers of the feature hierarchy and refines the spatial precision of the output. See Figure 3.
While fully convolutionalized classifiers fine-tuned to semantic segmentation both recognize and localize, as shown in Section 4.1, these networks can be improved to make direct use of shallower, more local features. Even though these base networks score highly on the standard metrics, their output is dissatisfyingly coarse (see Figure 4). The stride of the network prediction limits the scale of detail in the upsampled output.
We address this by adding skips  that fuse layer outputs, in particular to include shallower layers with finer strides in prediction. This turns a line topology into a DAG: edges skip ahead from shallower to deeper layers. It is natural to make more local predictions from shallower layers since their receptive fields are smaller and see fewer pixels. Once augmented with skips, the network makes and fuses predictions from several streams that are learned jointly and end-to-end.
Combining fine layers and coarse layers lets the model make local predictions that respect global structure. This crossing of layers and resolutions is a learned, nonlinear counterpart to the multi-scale representation of the Laplacian pyramid . By analogy to the jet of Koenderick and van Doorn , we call our feature hierarchy the deep jet.
Layer fusion is essentially an elementwise operation. However, the correspondence of elements across layers is complicated by resampling and padding. Thus, in general, layers to be fused must be aligned by scaling and cropping. We bring two layers into scale agreement by upsampling the lower-resolution layer, doing so in-network as explained in Section3.3. Cropping removes any portion of the upsampled layer which extends beyond the other layer due to padding. This results in layers of equal dimensions in exact alignment. The offset of the cropped region depends on the resampling and padding parameters of all intermediate layers. Determining the crop that results in exact correspondence can be intricate, but it follows automatically from the network definition (and we include code for it in Caffe).
Having spatially aligned the layers, we next pick a fusion operation. We fuse features by concatenation, and immediately follow with classification by a “score layer” consisting of a convolution. Rather than storing concatenated features in memory, we commute the concatenation and subsequent classification (as both are linear). Thus, our skips are implemented by first scoring each layer to be fused by convolution, carrying out any necessary interpolation and alignment, and then summing the scores. We also considered max fusion, but found learning to be difficult due to gradient switching. The score layer parameters are zero-initialized when a skip is added, so that they do not interfere with existing predictions of other streams. Once all layers have been fused, the final prediction is then upsampled back to image resolution.
Skip Architectures for Segmentation We define a skip architecture to extend FCN-VGG16 to a three-stream net with eight pixel stride shown in Figure 3. Adding a skip from pool4 halves the stride by scoring from this stride sixteen layer. The interpolation layer of the skip is initialized to bilinear interpolation, but is not fixed so that it can be learned as described in Section 3.3. We call this two-stream net FCN-16s, and likewise define FCN-8s by adding a further skip from pool3 to make stride eight predictions. (Note that predicting at stride eight does not significantly limit the maximum achievable mean IU; see Section 6.3.)
We experiment with both staged training and all-at-once training. In the staged version, we learn the single-stream FCN-32s, then upgrade to the two-stream FCN-16s and continue learning, and finally upgrade to the three-stream FCN-8s and finish learning. At each stage the net is learned end-to-end, initialized with the parameters of the earlier net. The learning rate is dropped from FCN-32s to FCN-16s and more from FCN-16s to FCN-8s, which we found to be necessary for continued improvements.
Learning all-at-once rather than in stages gives nearly equivalent results, while training is faster and less tedious. However, disparate feature scales make naïve training prone to divergence. To remedy this we scale each stream by a fixed constant, for a similar in-network effect to the staged learning rate adjustments. These constants are picked to approximately equalize average feature norms across streams. (Other normalization schemes should have similar effect.)
With FCN-16s validation score improves to 65.0 mean IU, and FCN-8s brings a minor improvement to 65.5. At this point our fusion improvements have met diminishing returns, so we do not continue fusing even shallower layers.
To identify the contribution of the skips we compare scoring from the intermediate layers in isolation, which results in poor performance, or dropping the learning rate without adding skips, which gives negligible improvement in score without refining the visual quality of output. All skip comparisons are reported in Table III. Figure 4 shows the progressively finer structure of the output.
Fine-tuning We fine-tune all layers by backpropagation through the whole net. Fine-tuning the output classifier alone yields only 73% of the full fine-tuning performance as compared in Table III. Fine-tuning in stages takes 36 hours on a single GPU. Learning FCN-8s all-at-once takes half the time to reach comparable accuracy. Training from scratch gives substantially lower accuracy.
More training data The PASCAL VOC 2011 segmentation training set labels 1,112 images. Hariharan et al.  collected labels for a larger set of 8,498 PASCAL training images, which was used to train the previous best system, SDS . This training data improves the FCN-32s validation score555There are training images from  included in the PASCAL VOC 2011 val set, so we validate on the non-intersecting set of 736 images. from 57.7 to 63.6 mean IU and improves the FCN-AlexNet score from 39.8 to 48.0 mean IU.
Loss The per-pixel, unnormalized softmax loss is a natural choice for segmenting images of any size into disjoint classes, so we train our nets with it. The softmax operation induces competition between classes and promotes the most confident prediction, but it is not clear that this is necessary or helpful. For comparison, we train with the sigmoid cross-entropy loss and find that it gives similar results, even though it normalizes each class prediction independently.
Patch sampling As explained in Section 3.4, our whole image training effectively batches each image into a regular grid of large, overlapping patches. By contrast, prior work randomly samples patches over a full dataset [10, 11, 12, 13, 16]
, potentially resulting in higher variance batches that may accelerate convergence
. We study this tradeoff by spatially sampling the loss in the manner described earlier, making an independent choice to ignore each final layer cell with some probability. To avoid changing the effective batch size, we simultaneously increase the number of images per batch by a factor . Note that due to the efficiency of convolution, this form of rejection sampling is still faster than patchwise training for large enough values of (e.g., at least for according to the numbers in Section 3.1). Figure 5 shows the effect of this form of sampling on convergence. We find that sampling does not have a significant effect on convergence rate compared to whole image training, but takes significantly more time due to the larger number of images that need to be considered per batch. We therefore choose unsampled, whole image training in our other experiments.
Class balancing Fully convolutional training can balance classes by weighting or sampling the loss. Although our labels are mildly unbalanced (about are background), we find class balancing unnecessary.
Dense Prediction The scores are upsampled to the input dimensions by backward convolution layers within the net. Final layer backward convolution weights are fixed to bilinear interpolation, while intermediate upsampling layers are initialized to bilinear interpolation, and then learned. This simple, end-to-end method is accurate and fast.
Augmentation We tried augmenting the training data by randomly mirroring and “jittering” the images by translating them up to 32 pixels (the coarsest scale of prediction) in each direction. This yielded no noticeable improvement.
We test our FCN on semantic segmentation and scene parsing, exploring PASCAL VOC, NYUDv2, SIFT Flow, and PASCAL-Context. Although these tasks have historically distinguished between objects and regions, we treat both uniformly as pixel prediction. We evaluate our FCN skip architecture on each of these datasets, and then extend it to multi-modal input for NYUDv2 and multi-task prediction for the semantic and geometric labels of SIFT Flow. All experiments follow the same network architecture and optimization settings decided on in Section 4.
Metrics We report metrics from common semantic segmentation and scene parsing evaluations that are variations on pixel accuracy and region intersection over union (IU):
frequency weighted IU:
where is the number of pixels of class predicted to belong to class , there are different classes, and is the total number of pixels of class .
PASCAL VOC Table IV gives the performance of our FCN-8s on the test sets of PASCAL VOC 2011 and 2012, and compares it to the previous best, SDS , and the well-known R-CNN . We achieve the best results on mean IU by 30% relative. Inference time is reduced (convnet only, ignoring proposals and refinement) or (overall).
NYUDv2  is an RGB-D dataset collected using the Microsoft Kinect. It has 1,449 RGB-D images, with pixelwise labels that have been coalesced into a 40 class semantic segmentation task by Gupta et al. . We report results on the standard split of 795 training images and 654 testing images. Table V gives the performance of several net variations. First we train our unmodified coarse model (FCN-32s) on RGB images. To add depth information, we train on a model upgraded to take four-channel RGB-D input (early fusion). This provides little benefit, perhaps due to similar number of parameters or the difficulty of propagating meaningful gradients all the way through the net. Following the success of Gupta et al. , we try the three-dimensional HHA encoding of depth and train a net on just this information. To effectively combine color and depth, we define a “late fusion” of RGB and HHA that averages the final layer scores from both nets and learn the resulting two-stream net end-to-end. This late fusion RGB-HHA net is the most accurate.
SIFT Flow is a dataset of 2,688 images with pixel labels for 33 semantic classes (“bridge”, “mountain”, “sun”), as well as three geometric classes (“horizontal”, “vertical”, and “sky”). An FCN can naturally learn a joint representation that simultaneously predicts both types of labels. We learn a two-headed version of FCN-32/16/8s with semantic and geometric prediction layers and losses. This net performs as well on both tasks as two independently trained nets, while learning and inference are essentially as fast as each independent net by itself. The results in Table VI, computed on the standard split into 2,488 training and 200 test images,666Three of the SIFT Flow classes are not present in the test set. We made predictions across all 33 classes, but only included classes actually present in the test set in our evaluation. show better performance on both tasks.
|mean IU||mean IU||inference|
|VOC2011 test||VOC2012 test||time|
|SDS ||52.6||51.6||50 s|
|Gupta et al. ||60.3||-||28.6||47.0|
|Liu et al. ||76.7||-||-||-||-|
|Tighe et al.  transfer||-||-||-||-||90.8|
|Tighe et al.  SVM||75.6||41.1||-||-||-|
|Tighe et al.  SVM+MRF||78.6||39.2||-||-||-|
|Farabet et al.  natural||72.3||50.8||-||-||-|
|Farabet et al.  balanced||78.5||29.6||-||-||-|
|Pinheiro et al. ||77.7||29.8||-||-||-|
PASCAL-Context  provides whole scene annotations of PASCAL VOC 2010. While there are 400+ classes, we follow the 59 class task defined by  that picks the most frequent classes. We train and evaluate on the training and val sets respectively. In Table VII we compare to the previous best result on this task. FCN-8s scores 39.1 mean IU for a relative improvement of more than 10%.
|FCN-8s||SDS ||Ground Truth||Image|
We examine the learning and inference of fully convolutional networks. Masking experiments investigate the role of context and shape by reducing the input to only foreground, only background, or shape alone. Defining a “null” background model checks the necessity of learning a background classifier for semantic segmentation. We detail an approximation between momentum and batch size to further tune whole image learning. Finally, we measure bounds on task accuracy for given output resolutions to show there is still much to improve.
Given the large receptive field size of an FCN, it is natural to wonder about the relative importance of foreground and background pixels in the prediction. Is foreground appearance sufficient for inference, or does the context influence the output? Conversely, can a network learn to recognize a class by its shape and context alone?
Masking To explore these issues we experiment with masked versions of the standard PASCAL VOC segmentation challenge. We both mask input to networks trained on normal PASCAL, and learn new networks on the masked PASCAL. See Table VIII for masked results.
Masking the foreground at inference time is catastrophic. However, masking the foreground during learning yields a network capable of recognizing object segments without observing a single pixel of the labeled class. Masking the background has little effect overall but does lead to class confusion in certain cases. When the background is masked during both learning and inference, the network unsurprisingly achieves nearly perfect background accuracy; however certain classes are more confused. All-in-all this suggests that FCNs do incorporate context even though decisions are driven by foreground pixels.
To separate the contribution of shape, we learn a net restricted to the simple input of foreground/background masks. The accuracy in this shape-only condition is lower than when only the foreground is masked, suggesting that the net is capable of learning context to boost recognition. Nonetheless, it is surprisingly accurate. See Figure 7.
Background modeling It is standard in detection and semantic segmentation to have a background model. This model usually takes the same form as the models for the classes of interest, but is supervised by negative instances. In our experiments we have followed the same approach, learning parameters to score all classes including background. Is this actually necessary, or do class models suffice?
To investigate, we define a net with a “null” background model that gives a constant score of zero. Instead of training with the softmax loss, which induces competition by normalizing across classes, we train with the sigmoid cross-entropy loss, which independently normalizes each score. For inference each pixel is assigned the highest scoring class. In all other respects the experiment is identical to our FCN-32s on PASCAL VOC. The null background net scores 1 point lower than the reference FCN-32s and a control FCN-32s trained on all classes including background with the sigmoid cross-entropy loss. To put this drop in perspective, note that discarding the background model in this way reduces the total number of parameters by less than 0.1%. Nonetheless, this result suggests that learning a dedicated background model for semantic segmentation is not vital.
In comparing optimization schemes for FCNs, we find that “heavy” online learning with high momentum trains more accurate models in less wall clock time (see Section 4.2). Here we detail a relationship between momentum and batch size that motivates heavy learning.
By writing the updates computed by gradient accumulation as a non-recursive sum, we will see that momentum and batch size can be approximately traded off, which suggests alternative training parameters. Let be the step taken by minibatch SGD with momentum at time ,
where is the loss for example and parameters , is the momentum, is the batch size, and is the learning rate. Expanding this recurrence as an infinite sum with geometric coefficients, we have
In other words, each example is included in the sum with coefficient , where the index orders the examples from most recently considered to least recently considered. Approximating this expression by dropping the floor, we see that learning with momentum and batch size appears to be similar to learning with momentum and batch size if . Note that this is not an exact equivalence: a smaller batch size results in more frequent weight updates, and may make more learning progress for the same number of gradient computations. For typical FCN values of momentum and a batch size of 20 images, an approximately equivalent training regime uses momentum and a batch size of one, resulting in online learning. In practice, we find that online learning works well and yields better FCN models in less wall clock time.
FCNs achieve good performance on the mean IU segmentation metric even with spatially coarse semantic prediction. To better understand this metric and the limits of this approach with respect to it, we compute approximate upper bounds on performance with prediction at various resolutions. We do this by downsampling ground truth images and then upsampling back to simulate the best results obtainable with a particular downsampling factor. The following table gives the mean IU on a subset55footnotemark: 5 of PASCAL 2011 val for various downsampling factors.
Pixel-perfect prediction is clearly not necessary to achieve mean IU well above state-of-the-art, and, conversely, mean IU is a not a good measure of fine-scale accuracy. The gaps between oracle and state-of-the-art accuracy at every stride suggest that recognition and not resolution is the bottleneck for this metric.
Fully convolutional networks are a rich class of models that address many pixelwise tasks. FCNs for semantic segmentation dramatically improve accuracy by transferring pre-trained classifier weights, fusing different layer representations, and learning end-to-end on whole images. End-to-end, pixel-to-pixel operation simultaneously simplifies and speeds up learning and inference. All code for this paper is open source in Caffe, and all models are freely available in the Caffe Model Zoo. Further works have demonstrated the generality of fully convolutional networks for a variety of image-to-image tasks.
This work was supported in part by DARPA’s MSEE and SMISC programs, NSF awards IIS-1427425, IIS-1212798, IIS-1116411, and the NSF GRFP, Toyota, and the Berkeley Vision and Learning Center. We gratefully acknowledge NVIDIA for GPU donation. We thank Bharath Hariharan and Saurabh Gupta for their advice and dataset tools. We thank Sergio Guadarrama for reproducing GoogLeNet in Caffe. We thank Jitendra Malik for his helpful comments. Thanks to Wei Liu for pointing out an issue wth our SIFT Flow mean IU computation and an error in our frequency weighted mean IU formula.
D. C. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Deep neural networks segment neuronal membranes in electron microscopy images.” inNIPS, 2012, pp. 2852–2860.
G. Papandreou, L.-C. Chen, K. Murphy, and A. L. Yuille, “Weakly-and semi-supervised learning of a DCNN for semantic image segmentation,” inICCV, 2015.
S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr, “Conditional random fields as recurrent neural networks,” inICCV, 2015.