Geometric variations due to scale, pose, viewpoint and part deformation present a major challenge in object recognition and detection. The current state-of-the-art method for addressing this issue is Deformable Convolutional Networks (DCNv1) , which introduces two modules that aid CNNs in modeling such variations. One of these modules is deformable convolution, in which the grid sampling locations of standard convolution are each offset by displacements learned with respect to the preceding feature maps. The other is deformable RoIpooling, where offsets are learned for the bin positions in RoIpooling 
. The incorporation of these modules into a neural network gives it the ability to adapt its feature representation to the configuration of an object, specifically by deforming its sampling and pooling patterns to fit the object’s structure. With this approach, large improvements in object detection accuracy are obtained.
Towards understanding Deformable ConvNets, the authors visualized the induced changes in receptive field, via the arrangement of offset sampling positions in PASCAL VOC images . It is found that samples for an activation unit tend to cluster around the object on which it lies. However, the coverage over an object is inexact, exhibiting a spread of samples beyond the area of interest. In a deeper analysis of spatial support using images from the more challenging COCO dataset , we observe that such behavior becomes more pronounced. These findings suggest that greater potential exists for learning deformable convolutions.
In this paper, we present a new version of Deformable ConvNets, called Deformable ConvNets v2 (DCNv2), with enhanced modeling power for learning deformable convolutions. This increase in modeling capability comes in two complementary forms. The first is the expanded use of deformable convolution layers within the network. Equipping more convolutional layers with offset learning capacity allows DCNv2 to control sampling over a broader range of feature levels. The second is a modulation mechanism in the deformable convolution modules, where each sample not only undergoes a learned offset, but is also modulated by a learned feature amplitude. The network module is thus given the ability to vary both the spatial distribution and the relative influence of its samples.
To fully exploit the increased modeling capacity of DCNv2, effective training is needed. Inspired by work on knowledge distillation in neural networks [2, 22], we make use of a teacher network for this purpose, where the teacher provides guidance during training. We specifically utilize R-CNN  as the teacher. Since it is a network trained for classification on cropped image content, R-CNN learns features unaffected by irrelevant information outside the region of interest. To emulate this property, DCNv2 incorporates a feature mimicking loss into its training, which favors learning of features consistent to those of R-CNN. In this way, DCNv2 is given a strong training signal for its enhanced deformable sampling.
With the proposed changes, the deformable modules remain lightweight and can easily be incorporated into existing network architectures. Specifically, we incorporate DCNv2 into the Faster R-CNN  and Mask R-CNN  systems, with a variety of backbone networks. Extensive experiments on the COCO benchmark demonstrate the significant improvement of DCNv2 over DCNv1 for object detection and instance segmentation. The code for DCNv2 will be released.
2 Analysis of Deformable ConvNet Behavior
2.1 Spatial Support Visualization
To better understand the behavior of Deformable ConvNets, we visualize the spatial support of network nodes by their effective receptive fields , effective sampling locations, and error-bounded saliency regions. These three modalities provide different and complementary perspectives on the underlying image regions that contribute to a node’s response.
Effective receptive fields Not all pixels within the receptive field of a network node contribute equally to its response. The differences in these contributions are represented by an effective receptive field, whose values are calculated as the gradient of the node response with respect to intensity perturbations of each image pixel . We utilize the effective receptive field to examine the relative influence of individual pixels on a network node, but note that this measure does not reflect the structured influence of full image regions.
Effective sampling / bin locations In , the sampling locations of (stacked) convolutional layers and the sampling bins in RoIpooling layers are visualized for understanding the behavior of Deformable ConvNets. However, the relative contributions of these sampling locations to the network node are not revealed. We instead visualize effective sampling locations that incorporate this information, computed as the gradient of the network node with respect to the sampling / bin locations, so as to understand their contribution strength.
Error-bounded saliency regions The response of a network node will not change if we remove image regions that do not influence it, as demonstrated in recent research on image saliency [41, 44, 13, 7]. Based on this property, we can determine a node’s support region as the smallest image region giving the same response as the full image, within a small error bound. We refer to this as the error-bounded saliency region, which can be found by progressively masking parts of the image and computing the resulting node response, as described in more detail in the Appendix. The error-bounded saliency region facilitates comparison of support regions from different networks.
2.2 Spatial Support of Deformable ConvNets
We analyze the visual support regions of Deformable ConvNets in object detection. The regular ConvNet we employ as a baseline consists of a Faster R-CNN + ResNet-50  object detector with aligned RoIpooling111Aligned RoIpooling is called RoIAlign in . We use the term “aligned RoIpooling” in this paper to more clearly describe it in the context of other related terms. 
. All the convolutional layers in ResNet-50 are applied on the whole input image. The effective stride in the conv5 stage is reduced from 32 to 16 pixels to increase feature map resolution. The RPN head is added on top of the conv4 features of ResNet-101. On top of the conv5 features we add the Fast R-CNN head , which is composed of aligned RoIpooling and two fully-connected (fc) layers, followed by the classification and bounding box regression branches. We follow the procedure in  to turn the object detector into its deformable counterpart. The three layers of convolutions in the conv5 stage are replaced by deformable convolution layers. Also, the aligned RoIpooling layer is replaced by deformable RoIPooling. Both networks are trained and visualized on the COCO benchmark. It is worth mentioning that when the offset learning rate is set to zero, the Deformable Faster R-CNN detector degenerates to regular Faster R-CNN with aligned RoIpooling.
Using the three visualization modalities, we examine the spatial support of nodes in the last layer of the conv5 stage in Figure 1 (a)(b). The sampling locations analyzed in  are also shown. From these visualizations, we make the following observations:
1. Regular ConvNets can model geometric variations to some extent, as evidenced by the changes in spatial support with respect to image content. Thanks to the strong representation power of deep ConvNets, the network weights are learned to accommodate some degree of geometric transformation.
2. By introducing deformable convolution, the network’s ability to model geometric transformation is considerably enhanced, even on the challenging COCO benchmark. The spatial support adapts much more to image content, with nodes on the foreground having support that covers the whole object, while nodes on the background have expanded support that encompasses greater context. However, the range of spatial support may be inexact, with the effective receptive field and error-bounded saliency region of a foreground node including background areas irrelevant for detection.
3. The three presented types of spatial support visualizations are more informative than the sampling locations used in . This can be seen, for example, with regular ConvNets, which have fixed sampling locations along a grid, but actually adapt its effective spatial support via network weights. The same is true for Deformable ConvNets, whose predictions are jointly affected by learned offsets and network weights. Examining sampling locations alone, as done in , can result in misleading conclusions about Deformable ConvNets.
Figure 2 (a)(b) display the spatial support of the 2fc node in the per-RoI detection head, which is directly followed by the classification and the bounding box regression branches. The visualization of effective bin locations suggests that bins on the object foreground generally receive larger gradients from the classification branch, and thus exert greater influence on prediction. This observation holds for both aligned RoIpooling and Deformable RoIpooling. In Deformable RoIpooling, a much larger proportion of bins cover the object foreground than in aligned RoIpooling, thanks to the introduction of learnable bin offsets. Thus, more information from relevant bins is available for the downstream Fast R-CNN head. Meanwhile, the error-bounded saliency regions in both aligned RoIpooling and Deformable RoIpooling are not fully focused on the object foreground, which suggests that image content outside of the RoI affects the prediction result. According to a recent study , such feature interference could be harmful for detection.
While it is evident that Deformable ConvNets have markedly improved ability to adapt to geometric variation in comparison to regular ConvNets, it can also be seen that their spatial support may extend beyond the region of interest. We thus seek to upgrade Deformable ConvNets so that they can better focus on pertinent image content and deliver greater detection accuracy.
3 More Deformable ConvNets
To improve the network’s ability to adapt to geometric variations, we present changes to boost its modeling power and to help it take advantage of this increased capability.
3.1 Stacking More Deformable Conv Layers
Encouraged by the observation that Deformable ConvNets can effectively model geometric transformation on challenging benchmarks, we boldly replace more regular conv layers by their deformable counterparts. We expect that by stacking more deformable conv layers, the geometric transformation modeling capability of the entire network can be further strengthened.
In this paper, deformable convolutions are applied in all the conv layers in stages conv3, conv4, and conv5 in ResNet-50. Thus, there are 12 layers of deformable convolution in the network. In contrast, just three layers of deformable convolution are used in , all in the conv5 stage. It is observed in  that performance saturates when stacking more than three layers for the relatively simple and small-scale PASCAL VOC benchmark. Also, misleading offset visualizations on COCO may have hindered further exploration on more challenging benchmarks. In experiments, we observe that utilizing deformable layers in the conv3-conv5 stages achieves the best tradeoff between accuracy and efficiency for object detection on COCO. See Section 5.2 for details.
3.2 Modulated Deformable Modules
To further strengthen the capability of Deformable ConvNets in manipulating spatial support regions, a modulation mechanism is introduced. With it, the Deformable ConvNets modules can not only adjust offsets in perceiving input features, but also modulate the input feature amplitudes from different spatial locations / bins. In the extreme case, a module can decide not to perceive signals from a particular location / bin by setting its feature amplitude to zero. Consequently, image content from the corresponding spatial location will have considerably reduced or no impact on the module output. Thus, the modulation mechanism provides the network module another dimension of freedom to adjust its spatial support regions.
Given a convolutional kernel of sampling locations, let and denote the weight and pre-specified offset for the -th location, respectively. For example, and defines a convolutional kernel of dilation 1. Let and denote the features at location from the input feature maps and output feature maps , respectively. The modulated deformable convolution can then be expressed as
where and are the learnable offset and modulation scalar for the -th location, respectively. The modulation scalar lies in the range , while is a real number with unconstrained range. As
is fractional, bilinear interpolation is applied as in in computing . Both and are obtained via a separate convolution layer applied over the same input feature maps . This convolutional layer is of the same spatial resolution and dilation as the current convolutional layer. The output is of channels, where the first channels correspond to the learned offsets , and the remaining channels are further fed to a sigmoid layer to obtain the modulation scalars . The kernel weights in this separate convolution layer are initialized to zero. Thus, the initial values of and are 0 and 0.5, respectively. The learning rates of the added conv layers for offset and modulation learning are set to 0.1 times those of the existing layers.
The design of modulated deformable RoIpooling is similar. Given an input RoI, RoIpooling divides it into spatial bins (). Within each bin, sampling grids of even spatial intervals are applied (). The sampled values on the grids are averaged to compute the bin output. Let and be the learnable offset and modulation scalar for the -th bin. The output binning feature is computed as
where is the sampling location for the -th grid cell in the -th bin, and denotes the number of sampled grid cells. Bilinear interpolation is applied to obtain features . The values of and are produced by a sibling branch on the input feature maps. In this branch, RoIpooling generates features on the RoI, followed by two fc
layers of 1024-D (initialized with Gaussian distribution of standard derivation of 0.01). On top of that, an additionalfc layer produces output of channels (weights initialized to be zero). The first channels are the normalized learnable offsets, where element-wise multiplications with the RoI’s width and height are computed to obtain . The remaining channels are normalized by a sigmoid layer to produce . The learning rates of the added fc layers for offset learning are the same as those of the existing layers.
3.3 R-CNN Feature Mimicking
As observed in Figure 2, the error-bounded saliency region of a per-RoI classification node can stretch beyond the RoI for both regular ConvNets and Deformable ConvNets. Image content outside of the RoI may thus affect the extracted features and consequently degrade the final results of object detection.
In , the authors find redundant context to be a plausible source of detection error for Faster R-CNN. Together with other motivations (, to share fewer features between the classification and bounding box regression branches), the authors propose to combine the classification scores of Faster R-CNN and R-CNN to obtain the final detection score. Since R-CNN classification scores are focused on cropped image content from the input RoI, incorporating them would help to alleviate the redundant context problem and improve detection accuracy. However, the combined system is slow because both the Faster-RCNN and R-CNN branches need to be applied in both training and inference.
Meanwhile, Deformable ConvNets are powerful in adjusting spatial support regions. For Deformable ConvNets v2 in particular, the modulated deformable RoIpooling module could simply set the modulation scalars of bins in a way that excludes redundant context. However, our experiments in Section 5.3 show that even with modulated deformable modules, such representations cannot be learned well through the standard Faster R-CNN training procedure. We suspect that this is because the conventional Faster R-CNN training loss cannot effectively drive the learning of such representations. Additional guidance is needed to steer the training.
, we incorporate a feature mimic loss on the per-RoI features of Deformable Faster R-CNN to force them to be similar to R-CNN features extracted from cropped images. This auxiliary training objective is intended to drive Deformable Faster R-CNN to learn more “focused” feature representations like R-CNN. We note that, based on the visualized spatial support regions in Figure2, a focused feature representation may well not be optimal for negative RoIs on the image background. For background areas, more context information may need to be considered so as not to produce false positive detections. Thus, the feature mimic loss is enforced only on positive RoIs that sufficiently overlap with ground-truth objects.
The network architecture for training Deformable Faster R-CNN is presented in Figure 3. In addition to the Faster R-CNN network, an additional R-CNN branch is added for feature mimicking. Given an RoI for feature mimicking, the image patch corresponding to it is cropped and resized to pixels. In the R-CNN branch, the backbone network operates on the resized image patch and produces feature maps of spatial resolution. A (modulated) deformable RoIpooling layer is applied on top of the feature maps, where the input RoI covers the whole resized image patch (top-left corner at , and height and width are 224 pixels). After that, 2 fc layers of 1024-D are applied, producing an R-CNN feature representation for the input image patch, denoted by . A
-way Softmax classifier follows for classification, wheredenotes the number of foreground categories, plus one for background. The feature mimic loss is enforced between the R-CNN feature representation and the counterpart in Faster R-CNN, , which is also 1024-D and is produced by the 2 fc
layers in the Fast R-CNN head. The feature mimic loss is defined on the cosine similarity betweenand , computed as
where denotes the set of RoIs sampled for feature mimic training. In the SGD training, given an input image, 32 positive region proposals generated by RPN are randomly sampled into . A cross-entropy classification loss is enforced on the R-CNN classification head, also computed on the RoIs in . Network training is driven by the feature mimic loss and the R-CNN classification loss, together with the original loss terms in Faster R-CNN. The loss weights of the two newly introduced loss terms are 0.1 times those of the original Faster R-CNN loss terms. The network parameters between the corresponding modules in the R-CNN and the Faster R-CNN branches are shared, including the backbone network, (modulated) deformable RoIpooling, and the 2 fc heads (the classification heads in the two branches are unshared). In inference, only the Faster R-CNN network is applied on the test images, without the auxiliary R-CNN branch. Thus, no additional computation is introduced by R-CNN feature mimicking in inference.
4 Related Work
is a long-standing problem in computer vision, and there has been tremendous effort in designing translation-invariant features. Prior to the deep learning era, notable works include scale-invariant feature transform (SIFT), oriented FAST and rotated BRIEF (ORB) , and deformable part-based models (DPM) 
. Such works are limited by the inferior representation power of handcrafted features and the constrained family of geometric transformations they address (, affine transformations). Spatial transformer networks (STN) is the first work on learning translation-invariant features for deep CNNs. It learns to apply global affine transformations to warp feature maps, but such transformations inadequately model the more complex geometric variations encountered in many vision tasks. Instead of performing global parametric transformations and feature warping, Deformable ConvNets sample feature maps in a local and dense manner, via learnable offsets in the proposed deformable convolution and deformable RoIpooling modules. Deformable ConvNets is the first work to effectively model geometric transformations in complex vision tasks (, object detection and semantic segmentation) on challenging benchmarks.
Our work extends Deformable ConvNets by enhancing its modeling power and facilitating network training. This new version of Deformable ConvNets yields significant performance gains over the original model.
Relation Networks and Attention Modules
are first proposed in natural language processing[14, 15, 4, 36] and physical system modeling [3, 38, 23, 35, 10, 32]. An attention / relation module effects an individual element (, a word in a sentence) by aggregating features from a set of elements (, all the words in the sentence), where the aggregation weights are usually defined on feature similarities among the elements. They are powerful in capturing long-range dependencies and contextual information in these tasks. Recently, the concurrent works of  and  successfully extend relation networks and attention modules to the image domain, for modeling long-range object-object and pixel-pixel relations, respectively. In , a learnable region feature extractor is proposed, unifying the previous region feature extraction modules from the pixel-object relation perspective. A common issue with such approaches is that the aggregation weights and the aggregation operation need to be computed on the elements in a pairwise fashion, incurring heavy computation that is quadratic to the number of elements (, all the pixels in an image). Our developed approach can be perceived as a special attention mechanism where only a sparse set of elements have non-zero aggregation weights (, pixels from among all the image pixels). The attended elements are specified by the learnable offsets, and the aggregation weights are controlled by the modulation mechanism. The computational overhead is just linear to the number of elements, which is negligible compared to that of the entire network (See Table 1).
Spatial Support Manipulation.
For atrous convolution, the spatial support of convolutional layers has been enlarged by padding zeros in the convolutional kernels. The padding parameters are handpicked and predetermined. In active convolution , which is contemporary with Deformable ConvNets, convolutional kernel offsets are learned via back-propagation. But the offsets are static model parameters fixed after training and shared over different spatial locations. In a multi-path network for object detection , multiple RoIpooling layers are employed for each input RoI to better exploit multi-scale and context information. The multiple RoIpooling layers are centered at the input RoI, and are of different spatial scales. A common issue with these approaches is that the spatial support is controlled by static parameters and does not adapt to image content.
Effective Receptive Field and Salient Region. Towards better interpreting how a deep network functions, significant progress has been made in understanding which image regions contribute most to network prediction. Recent works on effective receptive fields  and salient regions [41, 44, 13, 7] reveal that only a small proportion of pixels in the theoretical receptive field contribute significantly to the final network prediction. The effective support region is controlled by the joint effect of network weights and sampling locations. Here we exploit the developed techniques to better understand the network behavior of Deformable ConvNets. The resulting observations guide and motivate us to improve over the original model.
Network Mimicking and Distillation are recently introduced techniques for model acceleration and compression. Given a large teacher model, a compact student model is trained by mimicking the teacher model output or feature responses on training images [2, 22, 28]. The hope is that the compact model can be better trained by distilling knowledge from the large model.
Here we employ a feature mimic loss to help the network learn features that reflect the object focus and classification power of R-CNN features. Improved accuracy is obtained and the visualized spatial supports corroborate this approach.
|method||setting (shorter side 1000)||Faster R-CNN||Mask R-CNN|
|regular (aligned RoIpooling)||34.7||19.3||39.5||45.3||51.3M||326.7G||36.6||32.2||39.5M||447.5G|
|dconv@c5 + dpool (DCNv1)||38.0||20.7||41.8||52.2||52.7M||328.2G||40.4||35.3||40.9M||449.0G|
|dconv@c3c5 + dpool||41.0||22.0||45.1||56.6||53.0M||331.8G||42.4||37.0||41.3M||452.5G|
|mdconv@c3c5 + mdpool||41.7||22.2||45.8||58.7||65.5M||346.2G||43.1||37.3||53.8M||461.1G|
|method||setting (shorter side 800)||Faster R-CNN||Mask R-CNN|
|regular (aligned RoIpooling)||35.6||18.2||40.3||48.7||51.3M||196.8G||37.8||33.4||39.5M||303.5G|
|dconv@c5 + dpool (DCNv1)||38.2||19.1||42.2||54.0||52.7M||198.9G||40.3||35.0||40.9M||304.9G|
|dconv@c3c5 + dpool||40.0||21.1||44.6||56.3||53.0M||201.2G||41.8||36.4||41.3M||307.2G|
|mdconv@c3c5 + mdpool||40.8||21.3||45.0||58.5||65.5M||214.7G||42.7||37.0||53.8M||320.3G|
5.1 Experiment Settings
Our models are trained on the 118k images of the COCO 2017 train set. In ablation, evaluation is done on the 5k images of the COCO 2017 validation set. We also evaluate performance on the 20k images of the COCO 2017 test-dev set. The standard mean average-precision scores at different box and mask IoUs are used for measuring object detection and instance segmentation accuracy, respectively.
Faster R-CNN and Mask R-CNN are chosen as the baseline systems. ImageNet pre-trained ResNet-50 is utilized as the backbone. The implementation of Faster R-CNN is the same as in Section 3.3. For Mask R-CNN, we follow the implementation in . To turn the networks into their deformable counterparts, the last set of regular conv layers (close to the output in the bottom-up computation) are replaced by (modulated) deformable conv layers. Aligned RoIpooling is replaced by (modulated) deformable RoIpooling. Specially for Mask R-CNN, the two aligned RoIpooling layers with and
bins are replaced by two (modulated) deformable RoIpooling layers with the same bin numbers. In R-CNN feature mimicking, the feature mimic loss is enforced on the RoI head for classification only (excluding that for mask estimation). For both systems, the choice of hyper-parameters follows the latest Detectron code base except for the image resolution, which is briefly presented here. In both training and inference, images are resized so that the shorter side is 1,000 pixels222The previous default setting in Detectron is 800 pixels. Ablation on input image resolution is present in Appendix.
. Anchors of 5 scales and 3 aspect ratios are utilized. 2k and 1k region proposals are generated at a non-maximum suppression threshold of 0.7 at training and inference respectively. In SGD training, 256 anchor boxes (of positive-negative ratio 1:1) and 512 region proposals (of positive-negative ratio 1:3) are sampled for backpropagating their gradients. In our experiments, the networks are trained on 8 GPUs with 2 images per GPU for 16 epochs. The learning rate is initialized to 0.02 and is divided by 10 at the 10-th and the 14-th epochs. The weight decay and the momentum parameters are set toand 0.9, respectively.
5.2 Enriched Deformation Modeling
The effects of enriched deformation modeling are examined from ablations shown in Table 1. The baseline with regular CNN modules obtains an AP score of 34.7% for Faster R-CNN, and AP and AP scores of 36.6% and 32.2% respectively for Mask R-CNN. To obtain a DCNv1 baseline, we follow the original Deformable ConvNets paper by replacing the last three layers of convolution in the conv5 stage and the aligned RoIpooling layer by their deformable counterparts. This DCNv1 baseline achieves an AP score of 38.0% for Faster R-CNN, and AP and AP scores of 40.4% and 35.3% respectively for Mask R-CNN. The deformable modules considerably improve accuracy as observed in .
By replacing more regular conv layers by their deformable counterparts, the accuracy of both Faster R-CNN and Mask R-CNN steadily improve, with gains between 2.0% and 3.0% for AP and AP scores when the conv layers in conv3-conv5 are replaced. No additional improvement is observed on the COCO benchmark by further replacing the regular conv layers in the conv2 stage. By upgrading the deformable modules to modulated deformable modules, we obtain further gains between 0.3% and 0.7% in AP and AP scores. In total, enriching the deformation modeling capability yields a 41.7% AP score on Faster R-CNN, which is 3.7% higher than that of the DCNv1 baseline. On Mask R-CNN, 43.1% AP and 37.3% AP scores are obtained with the enriched deformation modeling, which are respectively 2.7% and 2.0% higher than those of the DCNv1 baseline. Note that the added parameters and FLOPs for enriching the deformation modeling are minor compared to those of the overall networks.
Shown in Figure 1 (b)(c), the spatial support of the enriched deformable modeling exhibits better adaptation to image content compared to that of DCNv1.
Table 2 presents the results at input image resolution of 800 pixels, which follows the default setting in the Detectron code base. The same conclusion holds.
5.3 R-CNN Feature Mimicking
Ablations of the design choices in R-CNN feature mimicking are shown in Table 3. With the enriched deformation modeling, R-CNN feature mimicking further improves the AP and AP scores by about 1% to 1.4% in both the Faster R-CNN and Mask R-CNN systems. Mimicking features of positive boxes on the object foreground is found to be particularly effective, and the results when mimicking all the boxes or just negative boxes are much lower. As shown in Figure 2 (c)(d), feature mimicking can help the network features better focus on the object foreground, which is beneficial for positive boxes. For the negative boxes, the network tends to exploit more context information (see Figure 2), where feature mimicking would not be helpful.
We also apply R-CNN feature mimicking to regular ConvNets without any deformable layers. Almost no accuracy gains are observed. The visualized spatial support regions are shown in Figure 2 (e), which are not focused on the object foreground even with the auxiliary mimic loss. This is likely because it is beyond the representation capability of regular ConvNets to focus features on the object foreground, and thus this cannot be learned.
|setting||regions to mimic||Faster R-CNN||Mask R-CNN|
|mdconv35 + mdpool||None||41.7||43.1||37.3|
|FG & BG||42.1||43.4||37.6|
|backbone||method||Faster R-CNN||Mask R-CNN|
5.4 Application on Stronger Backbones
Results on stronger backbones, by replacing ResNet-50 with ResNet-101 and ResNext-101 , are presented in Table 4. For the entries of DCNv1, the regular conv layers in the conv5 stage are replaced by the deformable counterpart, and aligned RoIpooling is replaced by deformable RoIpooling. For the DCNv2 entries, all the conv layers in the conv3-conv5 stages are of modulated deformable convolution, and modulated deformable RoIpooling is used instead, with supervision from the R-CNN feature mimic loss. DCNv2 is found to outperform regular ConvNet and DCNv1 considerably on all the network backbones.
Despite the superior performance of Deformable ConvNets in modeling geometric variations, its spatial support extends well beyond the region of interest, causing features to be influenced by irrelevant image content. In this paper, we present a reformulation of Deformable ConvNets which improves its ability to focus on pertinent image regions, through increased modeling power and stronger training. Significant performance gains are obtained on the COCO benchmark for object detection and instance segmentation.
|method||setting||Faster R-CNN + ResNet-101|
|regular||single-scale, shorter side 800||39.1||60.6||42.2||20.5||42.9||51.3|
|single-scale, shorter side 1000 (best)||39.2||60.6||42.4||21.6||42.2||51.3|
|DCNv2||single-scale, shorter side 800||44.0||65.9||48.1||23.2||47.7||59.6|
|single-scale, shorter side 1000 (best)||44.8||66.3||48.8||24.4||48.1||59.6|
|method||offset&modulation pretraining||VOC det||VOC seg||ImageNet VID det||COCO det|
Appendix A1 Error-bounded Image Saliency
In existing research on image saliency [41, 44, 13, 7], a widely utilized formulation is as follows. Given an input image and a trained network , let denote the network response on the original image. A binary mask , which is of the same spatial dimension as , can be applied on the image as . For the image pixel where , its content is kept in the masked image. Meanwhile, if
, the content is set as 0 in the masked image. The saliency map is obtained by optimizing loss functionas a function of , where is the hyper-parameter balancing the output reconstruction error and the salient area loss . The optimized mask is called the saliency map. The problem is it is hard to obtain the salient region at a specified reconstruction error. Thus it is hard to compare the salient regions from two networks at the same reconstruction error.
We seek to strictly constrain the reconstruction loss in the image saliency formulation, so as to facilitate comparison among the salient regions derived from different networks. Thus, the optimization problem is slightly modified to be
where denotes an arbitrary form of reconstruction loss, which is strictly bounded by . We term the collection of image pixels where in the optimized mask as visual support region.
The formulation in Eq. (4
) is hard to be optimized, due to the hard reconstruction error constraint introduced. Here we develop a heuristic two-step procedure to reduce the search space in deriving the visual support region. At the first step, the visual support region is constrained to be rectangular of arbitrary shape. The rectangular is centered on the node to be interpreted. The rectangular is initialized of area size 0, and is enlarged gradually (at even area increment). The enlargement stops upon the reconstruction error constraint is satisfied. At the second step, pixel-level visual support region is derived within the rectangular area. The image is segmented into super-pixels by the algorithm in, so as to restrict the solution space. At initial, all the super-pixels within the rectangular are counted in the visual support region (taking mask value 1). Then the super-pixels are gradually removed in a greedy manner. At each iteration, the super-pixel causing the smallest rise in reconstruction error is removed. The iteration stops till the constraint would be violated by removing anymore super-pixels.
We apply the two-step procedure to visualize network nodes in Faster R-CNN object detector . We visualize both feature map nodes shared on the whole image, and the 2fc
node in the per-RoI detection head, which is directly followed by the classification and the bounding box regression branches. For image-wise feature map nodes (at a certain location), square rectangular is applied in the two-step procedure. For RoI-wise feature nodes, the rectangular is of the same aspect ratio as the input RoI. For both image-wise and RoI-wise nodes, the reconstruction loss is one minus the cosine similarity between the feature vectors derived from masked and original images. The error upper boundis set as 0.1.
Appendix A2 DCNv2 with Various Image Resolution
Figure 4 presents the results of applying regular ConvNets and DCNv2 on images of various resolutions. The baseline model is Faster R-CNN with ResNet-50  and ResNet-101. Models are trained and applied on images of shorter side pixels, respectively. DCNv2 is found to outperform regular ConvNet on all input resolutions. For DCNv2, the highest AP scores are obtained at input images of shorter side 1,000 pixels. With the shorter side larger than 1,000 pixels, AP scores of regular ConvNet decrease noticeably, while those of DCNv2 are almost unchanged. This phenomenon is more obvious for objects of large and medium sizes. As shown in Figure 5, the spatial support of regular ConvNets can just cover a small portion of the large objects at such high resolution, and the accuracy suffers. Meanwhile, the spatial support of DCNv2 can effectively adapt to objects at various resolutions.
Table 5 presents the results of multi-scale testing using ResNet-101. We first apply the DCNv2 model trained on the best single-scale setting (shorter side of 1000 pixels) on multi-scale testing images. Following the latest Detectron  code base, test images range from shorter side of 400 to 1400 pixels with step size of 200 pixels. Multi-scale testing of DCNv2 improves the AP score by 1.2% compared with the best single-scale setting.
Appendix A3 ImageNet Pre-Trained DCNv2
It is well known that many vision tasks benefit from ImageNet pre-training. This section investigates pre-training the learnable offsets and modulation scalars of DCNv2 on ImageNet , and finetuning on several tasks.
DCNv2 together with its DCNv1  and regular ConvNet counterparts are pretrained on the ImageNet-1K training set. In experiments, we follow  for the training and inference settings. In DCNv1, the conv layers in the conv5 stage are replaced by deformable conv layers. In DCNv2, all the conv layers in the conv3conv5 stages are replaced by modulated deformable conv layers.
Table 6 presents the top-1 and top-5 classification accuracies on the validation set. DCNv2 achieves noticeable improvements over both the regular and DCNv1 baselines, with minor additional computation overhead. The enriched deformation modeling capability of DCNv2 is beneficial for the ImageNet classification task itself.
Fine-tuning for Specific Tasks
We investigate the effect of ImageNet pretrained DCNv2 models on several tasks, including object detection on Pascal VOC, ImageNet VID and COCO, and semantic segmentation on Pascal VOC333Note the mimicking module is not involved in semantic segmentation experiments.. In experiments, Faster R-CNN and DeepLab-v1  are adopted as the baseline systems for object detection and semantic segmentation, respectively. For object detection on COCO, we follow the same settings as in Section 5.1. For experiments on Pascal VOC, we mainly follow  for the training and inference settings. Note that the baseline accuracy is higher than that reported in  mainly because of a better ImageNet pretrained model and the introduction of RoIAlign in object detection. For object detection on ImageNet VID, we mainly follow the protocol in [27, 43, 42] for the training and inference settings. The details are presented at the end of this section.
compares the performance of DCNv2 on various tasks using different pre-trained models. By pre-training the learnable offsets and modulation scalars on ImageNet, rather than initializing them as zeros prior to fine-tuning, noticeably accuracy improvements are observed on PASCAL VOC object detection and semantic segmentation. Meanwhile, the effect of pre-training on COCO detection is minor. This is probably because the larger and more challenging benchmark of COCO is sufficient for learning the offsets and the modulation scalars from scratch.
ImageNet VID settings. The models are trained on the union of the ImageNet VID training set and the ImageNet DET training set (only the same 30 category labels are used), and are evaluated on the ImageNet VID validation set. In both training and inference, the input images are resized to a shorter side of 600 pixels. In RPN, the anchors are of 3 aspect ratios 1:2, 1:1, 2:1 and 4 scales . 300 region proposals are generated for each frame at an NMS threshold of 0.7 IoU. SGD training is performed, with one image at each mini-batch. 120k iterations are performed on 4 GPUs, with each GPU holding one mini-batch. The learning rates are and in the ﬁrst 80k and last 40k iterations,respectively. In each mini-batch,images are sampled from ImageNet DET and ImageNet VID at a 1:1 ratio. The weight decay and the momentum parameters are set to 0.0001 and 0.9, respectively. In inference, detection boxes are generated at an NMS threshold of 0.3 IoU.
-  R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, et al. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274–2282, 2012.
-  J. Ba and R. Caruana. Do deep nets really need to be deep? In NIPS, 2014.
-  P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, et al. Interaction networks for learning about objects, relations and physics. In NIPS, 2016.
D. Britz, A. Goldie, M.-T. Luong, and Q. Le.
Massive exploration of neural machine translation architectures.In EMNLP, 2017.
-  L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915, 2016.
-  B. Cheng, Y. Wei, H. Shi, R. Feris, J. Xiong, and T. Huang. Revisiting rcnn: On awakening the classification power of faster rcnn. In ECCV, 2018.
-  P. Dabkowski and Y. Gal. Real time image saliency for black box classifiers. In NIPS, 2017.
-  J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In ICCV, 2017.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
-  M. Denil, S. G. Colmenarejo, S. Cabi, D. Saxton, and N. de Freitas. Programmable agents. arXiv preprint arXiv:1706.06383, 2017.
-  M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. IJCV, 2010.
-  P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 2010.
-  R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.
-  J. Gehring, M. Auli, D. Grangier, and Y. N. Dauphin. A convolutional encoder model for neural machine translation. In ACL, 2017.
-  J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017.
-  R. Girshick. Fast R-CNN. In ICCV, 2015.
-  R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
-  R. Girshick, I. Radosavovic, G. Gkioxari, P. Dollár, and K. He. Detectron. https://github.com/facebookresearch/detectron, 2018.
-  J. Gu, H. Hu, L. Wang, Y. Wei, and J. Dai. Learning region features for object detection. In ECCV, 2018.
-  K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In ICCV, 2017.
-  K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
-  G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. STAT, 2015.
-  Y. Hoshen. Vain: Attentional multi-agent predictive modeling. In NIPS, 2017.
-  H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei. Relation networks for object detection. In CVPR, 2018.
-  M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. In NIPS, 2015.
-  Y. Jeon and J. Kim. Active convolution: Learning the shape of convolution for image classification. In CVPR, 2017.
-  B. Lee, E. Erdenee, S. Jin, M. Y. Nam, Y. G. Jung, and P. K. Rhee. Multi-class multi-object tracking using changing point detection. In ECCV, 2016.
-  Q. Li, S. Jin, and J. Yan. Mimicking very efficient network for object detection. In CVPR, 2017.
-  T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
-  D. G. Lowe. Object recognition from local scale-invariant features. In ICCV, 1999.
-  W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive field in deep convolutional neural networks. arXiv preprint arXiv:1701.04128, 2017.
-  D. Raposo, A. Santoro, D. Barrett, R. Pascanu, T. Lillicrap, and P. Battaglia. Discovering objects and their relations from entangled scene representations. In ICLR, 2017.
-  S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
-  E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb: an efficient alternative to sift or surf. In ICCV, 2011.
-  A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. In NIPS, 2017.
-  A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017.
-  X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In CVPR, 2018.
-  N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks. In NIPS, 2017.
-  S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
-  S. Zagoruyko, A. Lerer, T.-Y. Lin, P. H. Pinheiro, S. Gross, S. Chintala, and P. Dollar. A multipath network for object detection. In BMVC, 2016.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.
Learning deep features for discriminative localization.In CVPR, 2016.
-  X. Zhu, Y. Wang, J. Dai, L. Yuan, and Y. Wei. Flow-guided feature aggregation for video object detection. In ICCV, 2017.
-  X. Zhu, Y. Xiong, J. Dai, L. Yuan, and Y. Wei. Deep feature flow for video recognition. In CVPR, 2017.
-  L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling. Visualizing deep neural network decisions: Prediction difference analysis. In ICLR, 2017.