Beyond Skip Connections: Top-Down Modulation for Object Detection

by   Abhinav Shrivastava, et al.

In recent years, we have seen tremendous progress in the field of object detection. Most of the recent improvements have been achieved by targeting deeper feedforward networks. However, many hard object categories such as bottle, remote, etc. require representation of fine details and not just coarse, semantic representations. But most of these fine details are lost in the early convolutional layers. What we need is a way to incorporate finer details from lower layers into the detection architecture. Skip connections have been proposed to combine high-level and low-level features, but we argue that selecting the right features from low-level requires top-down contextual information. Inspired by the human visual pathway, in this paper we propose top-down modulations as a way to incorporate fine details into the detection framework. Our approach supplements the standard bottom-up, feedforward ConvNet with a top-down modulation (TDM) network, connected using lateral connections. These connections are responsible for the modulation of lower layer filters, and the top-down network handles the selection and integration of contextual information and low-level features. The proposed TDM architecture provides a significant boost on the COCO testdev benchmark, achieving 28.6 AP for VGG16, 35.2 AP for ResNet101, and 37.3 for InceptionResNetv2 network, without any bells and whistles (e.g., multi-scale, iterative box refinement, etc.).


page 1

page 9


Learning to Refine Object Segments

Object segmentation requires both object-level information and low-level...

EDN: Salient Object Detection via Extremely-Downsampled Network

Recent progress on salient object detection (SOD) mainly benefits from m...

Attentional Local Contrast Networks for Infrared Small Target Detection

To mitigate the issue of minimal intrinsic features for pure data-driven...

LC3Net: Ladder context correlation complementary network for salient object detection

Currently, existing salient object detection methods based on convolutio...

Asymmetric Contextual Modulation for Infrared Small Target Detection

Single-frame infrared small target detection remains a challenge not onl...

Left-Right Skip-DenseNets for Coarse-to-Fine Object Categorization

Inspired by the recent neuroscience studies on the left-right asymmetry ...

1 Introduction

Figure 1: Detecting objects such as the bottle or remote shown above requires low-level finer details as well as high-level contextual information. In this paper, we propose a top-down modulation (TDM) network, which can be used with any bottom-up, feedforward ConvNet. We show that the features learnt by our approach lead to significantly improved object detection.

Convolutional neural networks (ConvNets) have revolutionized the field of object detection [17, 16, 41, 49, 14, 45, 4]. Most standard ConvNets are bottom-up, feedforward architectures constructed using repeated convolutional layers (with non-linearities) and pooling operations [28, 46, 24, 48, 47]. These convolutional layers learn invariances and the spatial pooling increases the receptive field of subsequent layers; thus resulting in a coarse, highly semantic representation at the final layer.

However, consider the images shown in Figure 1. Detecting and recognizing an object like the bottle in the left image or remote in the right image requires extraction of very fine details such as the vertical and horizontal parallel edges. But these are exactly the type of edges ConvNets try to gain invariance against in early convolutional layers. One can argue that ConvNets can learn not to ignore such edges when in context of other objects like table. However, objects such as table do not emerge until very late in feed-forward architecture. So, how can we incorporate these fine details in object detection?

A popular solution is to use variants of ‘skip’ connections [43, 10, 4, 22, 35, 53], that capture these finer details from lower convolutional layers with local

receptive fields. But simply incorporating high-dimensional skip features into detection does not yield significant improvements due to overfitting caused by curse of dimensionality. What we need is a selection/attention mechanism that selects the relevant features from lower convolutional layers.

We believe the answer lies in the process of top-down modulation. In the human visual pathway, once receptive field properties are tuned using feedforward processing, top-down modulations are evoked by feedback and horizontal connections [30, 27]. These connections modulate representations at multiple levels [54, 55, 13, 37, 15] and are responsible for their selective combination [25, 7]. We argue that the use of skip connections is a special case of this process, where the modulation

is relegated to the final classifier, which directly tries to influence lower layer features and/or learn how to combine them.

In this paper, we propose to incorporate the top-down modulation process in the ConvNet itself. Our approach supplements the standard bottom-up, feedforward ConvNet with a top-down network, connected using lateral connections. These connections are responsible for the modulation and selection of the lower layer filters, and the top-down network handles the integration of features.

Specifically, after a bottom-up ConvNet pass, the final high-level semantic features are transmitted back by the top-down network. Bottom-up features at intermediate depths, after lateral processing, are combined with the top-down features, and this combination is further transmitted down by the top-down network. Capacity of the new representation is determined by lateral and top-down connections, and optionally, the top-down connections can increase the spatial resolution of features. These final, possibly high-res, top-down features inherently have a combination of local and larger receptive fields.

The proposed Top-Down Modulation (TDM) network is trained end-to-end and can be readily applied to any base ConvNet architecture (, VGG [46], ResNet [24], Inception-Resnet [47] ). To demonstrate its effectiveness, we use the proposed network in the standard Faster R-CNN detection method [41] and evaluate on the challenging COCO benchmark [33]. We report a consistent and significant boost in performance on all metrics across network architectures. TDM network increases the performance of vanilla Faster R-CNN with: (a) VGG16 from 23.3 AP to textbf28.6 AP, (b) ResNet101 from 31.5 AP to 35.2 AP, and (c) InceptionResNetv2 from 34.7 AP to 37.2 AP. These are the best performances reported to-date for these architectures without any bells and whistles (, multi-scale features, iterative box-refinement). Furthermore, we see drastic improvements in small objects (, 4.5 AP) and in objects where selection of fine details using top-down context is important.

Figure 2: The illustration shows an example of Top-Down Modulation (TDM) Network, which is integrated with the bottom-up network with lateral connections. are bottom-up, feedforward feature blocks, are the lateral modules which transform low level features for the top-down contextual pathway. Finally, , which represent flow of top-down information from index to . Individual components are explained in Figure 3 and 4.

2 Related Work

After the resurgence of ConvNets for image classification [28, 8]

, they have been successfully adopted for a variety of computer vision tasks such as object detection 

[17, 16, 41, 49], semantic segmentation [6, 34, 35, 3], instance segmentation [21, 22, 38]

, pose estimation 

[51, 50], depth estimation [9, 52], edge detection [53], optical flow predictions [11, 40] . However, by construction, final ConvNet features lack the finer details that are captured by lower convolutional layers. These finer details are considered necessary for a variety of recognition tasks, such as accurate object localization and segmentation.

To counter this, ‘skip’ connections have been widely used with ConvNets. Though the specifics of methods vary widely, the underlying principle is same: using or combining finer features from lower layers and coarse semantic features for higher layers. For example, [43, 4, 10, 22] combine features from multiple layers for the final classifier; while [43, 4] use subsampled features from finer scales, [10, 22] upsample the features to the finest scale and use their combination. Instead of combining features, [35, 53, 34] do independent predictions at multiple layers and average the results. In our proposed framework, such upsampling, subsampling and fusion operations can be easily controlled by the lateral and top-down connections.

The proposed TDM network is conceptually similar to the strategies explored in other contemporary works [39, 40, 3, 32, 42]. All methods, including ours, propose architectures that go beyond the standard skip-connection paradigm and/or follow a coarse-to-fine strategy when using features from multiple levels of the bottom-up feature hierarchy. However, different methods focus on different tasks which guide their architectural design and training methodology.

Conv-deconv [36] and encoder-decoder style networks [3, 42] have been used for image segmentation to utilize finer features via lateral connections. These connections generally use the ‘unpool’ operation [56], which merely inverts the spatial pooling operation. Moreover, there is no modulation of bottom-up network. In comparison, our formulation is more generic and is responsible for the flow of high-level context features [37].

Pinheiro  [39] focus on refining class-agnostic object proposals by first selecting proposals purely based on bottom-up feedforward features [38], and then post-hoc learning how to refine each proposal independently using top-down and lateral connections (due to the computational complexity, only a few proposals can be selected to be refined). We argue that this use of top-down and lateral connections for refinement is sub-optimal for detection because it relies on the proposals selected based on feedforward features, which are insufficient to represent small and difficult objects. This training methodology also limits the ability to update the feedforward network through lateral connections. In contrast to this, we propose to learn better features for recognition tasks in an end-to-end trained system, and these features are used for both proposal generation and object detection. Similar to the idea of coarse-to-fine refinement, Ranjan and Black [40] propose a coarse-to-fine spatial pyramid network, which computes a low resolution residual optical flow and iteratively improves predictions with finer pyramid levels. This is akin to only using a specialized top-down network, which is suited for low-level tasks (like optical flow) but not for recognition tasks. Moreover, such purely coarse-to-fine networks cannot utilize models pre-trained on large-scale datasets [8], which is important for recognition tasks [17]. Therefore, our approach learns representation using bottom-up (fine-to-coarse), top-down (coarse-to-fine) and lateral networks simultaneously, and can use different pre-trained modules.

The proposed top-down network is closest to the recent work of Lin  [32], developed concurrently to ours, on feature pyramid network for object detection. Lin  [32] use bottom-up, top-down and lateral connections to learn a feature pyramid, and require multiple proposal generators and region classifiers on each level of the pyramid. In comparison, the proposed top-down modulation focuses on using these connections to learn a single final feature map that is used by a single proposal generator and region classifier.

There is strong evidence of such top-down context, feedback and lateral processing in the human visual pathway [30, 27, 15, 29, 54, 55, 13, 37, 25, 7]; wherein, the top-down signals are responsible for modulating low-level features [54, 55, 13, 37, 15] as well as act as attentional mechanism for selection of features [25, 7]. In this paper, we propose a computation model that captures some of these intuitions and incorporates them in a standard ConvNets, giving substantial performance improvements.

Our top-down framework is also related to the process of contextual feedback [2]. To incorporate top-down feedback loop in ConvNets, contemporary works [5, 12, 31, 44], have used ‘unrolled’ networks (trained stage-wise). The proposed top-down network with lateral connections explores a complementary paradigm and can be readily combined with them. Contextual features have also been used for ConvNets based object detectors; , using other objects [20] or regions [18] as context. We believe the proposed top-down path can naturally transmit these contextual features.

3 Top-Down Modulation (TDM)

Our goal is to incorporate top-down modulation into current object detection frameworks. The key idea is to select/attend to fine details from lower level feature maps based on top-down contextual features and select top-down contextual features based on the fine low-level details. We formalize this by proposing a simple top-down modulation (TDM) network as shown in Figure 2.

The TDM network starts from the last layer of bottom-up feedforward network. For example, in the case of VGG16, the input to the first layer of the TDM network is the conv5_3 output. Every layer of TDM network also gets the bottom-up features as inputs via lateral connections. Thus, the TDM network learns to: (a) transmit high-level contextual features that guide the learning and selection of relevant low-level features, and (b) use the bottom-up features to select the contextual information to transmit. The output of the proposed network captures both pertinent finer details and high-level information.

3.1 Proposed Architecture

An overview of the proposed framework is illustrated in Figure 2. The standard bottom-up network is represented by blocks of layers, where each block has multiple operations. The TDM network hinges on two key components: a lateral module , and a top-down module (see Figure 3). Each lateral module takes in a bottom-up feature (output of ) and produces the corresponding lateral feature . These lateral features and top-down features are combined, and optionally upsampled, by the module to produce the top-down features . These modules, and , control the capacity of the modulation network by changing their output feature dimensions.

The feature from the last top-down module is used for the task of object detection. For example, in Figure 2, instead of , we use as input to ROI proposal and ROI classifier networks of the Faster R-CNN [41] detection system (discussed in Section 4.1

). During training, gradient updates from the object detector backpropagate via top-down and lateral modules to the

blocks. The lateral modules learn how to transform low-level features and the top-down modules learn what semantic or context information to preserve in the top-down feature transmission as well as the selection of relevant low-level lateral features. Ultimately, the bottom-up features are modulated to adapt for this new representation.

Figure 3: The basic building blocks of Top-Down Modulation Network (detailed Section 3.1).

Architecture details.

The top-down and lateral modules described above are essentially small ConvNets, which can vary from a single or a hierarchy of convolutional layers to more involved blocks with Residual [24] or Inception [47] units. In this paper, we limit our study by using modules with a single convolutional layer and non-linearity to analyze the impact of top-down modulation.

A detailed example of lateral and top-down modules is illustrated in Figure 4. The lateral module is a

convolutional layer with ReLU non-linearity, which transforms an

input , to lateral feature . The top-down module is also a convolutional layer with ReLU, that combines this lateral feature with top-down feature , to produce an intermediate output . If the resolution of next lateral feature is higher than the previous lateral feature (, , then also upsamples the intermediate output to produce top-down feature . In Figure 2, we denote for simplicity. The final module can additionally have a convolutional layer with ReLU to output a feature, which is used by the detection system.

Varying , and

controls the capacity of the top-down modulation system and dimension of the output features. These hyperparameters are governed by the base network design, detection system and hardware constraints (discussed in Section 

4.3). Notice that the upsampling step is optional and depends on the content and arrangement of blocks (, no upsampling by in Figure 2). Also, the first top-down module ( in the illustration) only operates on (the final output of the bottom-up network).

Training methodology.

Integrating top-down modulation framework into a bottom-up ConvNet is only meaningful when the latter can represent high-level concepts in higher layers. Thus, we typically start with a pre-trained bottom-up network (see Section 6.4 for discussion). Starting with this pre-trained network, we find that progressively building the top-down network performs better in general. Therefore, we add one new pair of lateral and top-down modules at a time. For example, for the illustration in Figure 2, we will begin by adding and use to get features for object detection. After training modules, we will add the next pair and use a new module to get features for detection; and we will repeat this process. With each new pair, the entire network, top-down and bottom-up along with lateral connections, is trained end-to-end. Implementation details of this training methodology depends on the base network architecture, and will be discussed in Section 4.3.

To better understand the impact of the proposed TDM network, we conduct extensive experimental evaluation; and provide ablation analysis of various design decisions. We describe our approach in detail (including preliminaries and implementation details) in Section 4 and present our results in Section 5. We also report ablation analysis in Section 6. We would like to highlight that the proposed framework leads to substantial performance gains across different base network architectures, indicating its wide applicability.

Figure 4: An example with details of top-down modules and lateral connections. Please see Section 3.1 for details of the architecture.

4 Approach Details

name N
conv1_x 2
conv2_x 2
conv3_x 3
conv4_x 3
conv5_x 3
name NB N
conv1 1 1
conv2_x 3 9
conv3_x 4 12
conv4_x 23 69
name NI N
conv_x - 5
Mixed_5b 1 7
Block_10x 10 70
Mixed_6a 1 3
Block_20x 20 100
Table 1: Base networks architecture details for VGG16, ResNet101 and InceptionResNetv2. Legend: : bottom-up block id, N: number of convolutional filters, NR: number of residual units, NI: number of inception-resnet units, : Resolution and dimensions of the output feature. Refer to [46, 24, 47, 26] for details

In this section, we describe the preliminaries and provide implementation details of our top-down modulation (TDM) network under various settings. We first give a a brief overview of the object detection system and the ConvNet architectures used throughout this paper.

4.1 Preliminaries: Faster R-CNN

We use the Faster R-CNN [41] framework as our base object detection system. Faster R-CNN consists of two core modules: 1) ROI Proposal Network (RPN), which takes an image as input and proposes rectangular regions of interests (ROIs); and 2) ROI Classifier Network (RCN), which is a Fast R-CNN [16] detector that classifies these proposed regions and learns to refine ROI coordinates. Given an image, Faster R-CNN first uses a ConvNet to extract features that are shared by both RPN and RCN. RPN uses these features to propose candidate ROIs, which are then classified by RCN. The RCN network projects each ROI onto the shared feature map and performs the ‘ROI Pooling’ [16, 23] operation to extract a fixed length representation. Finally, this feature is used for classification and box regression. See [16, 23, 41, 26] for details.

Due to lack of support for recent ConvNet architectures [24, 47]

in the Faster R-CNN framework, we use our implementation in Tensorflow 

[1]. We follow the design choices outlined in [26]. In Section 5, we will provide performance numbers using both the released code [41] as well as our implementation (which tends to generally perform better). We use the end-to-end training paradigm for Faster R-CNN for all experiments [41]

. Unless specified otherwise, all methods start with models that were pre-trained on ImageNet classification 

[8, 17].

4.2 Preliminaries: Base Network Architectures

In this paper, we use three standard ConvNet architectures: VGG16 [46], ResNet101 [24] and InceptionResNetv2 [47]. We briefly explain how they are incorporated in the Faster R-CNN framework (see [41, 24, 26] for details), and give a quick overview of these architectures with reference to the bottom-up blocks from Section 3.1.

We use the term ‘Base Network’ to refer to the part of ConvNet that is shared by both RPN and RCN; and ‘Classifier Network’ to refer to the part that is used as RCN. For VGG16 [46, 41], ConvNet till conv5_3 is used as the base network, and the following two fc layers are used as the classifier network. Similarly, for ResNet101 [24, 41], base network is the ConvNet till conv4_x, and classifier network is the conv5_x block (with residual units or convolutional layers). For InceptionResNet101v2 [47, 26], ConvNet till the ‘Block_20x’ is used as the base network, and the remaining layers (‘Mixed_7a’ and ‘Block_9x’, with a total of 11 inception-resnet units or 48 convolutional layers) are used as the classifier network. Following [26]

, we change the pooling stride of the penultimate convolutional block in ResNet101 and InceptionResNetv2 to

to maintain spatial resolution, and use atrous [6, 34] convolution to recover the original field-of-view. Properties of bottom-up blocks , including number of layers, the output feature resolution and feature dimension are given in Table 1.

128 128 256
64 64 128
64 64 128
64 64 128
128 128 1024
128 128 1024
32 32 1024
576 512 1088
512 256 1088
Table 2: Top-Down Modulation network design for VGG16, ResNet101 and InceptionResNetv2. Notice that VGG16 is much smaller than 512, thus requiring fewer parameters in RPN and RCN modules. Also note that it is important to keep fixed for ResNet101 and InceptionResNetv2 in order to utilize pre-trained RPN and RCN modules

4.3 Top-Down Modulation

To add the proposed TDM network to the ConvNet architectures described above, we need to decide the extent of top-down modulation, the frequency of lateral connections and the capacity of , and modules. We try to follow these principles when making design decisions: (a) coarse semantic modules need larger capacity; (b) lateral and top-down connections should reduce feature dimensionality in order to force selection; and (c) the capacity of should be informed by the Proposal (RPN) and Classifier (RCN) Network design. Finally, the hardware constraint that a TDM augmented ConvNet should fit on a standard GPU.


Method train test ROINet ClsNet AP AP AP AP AP AP AR AR AR AR AR AR


Faster R-CNN [41] train val VGG16 VGG16 21.2 41.5 - - - - - - - - - -
Faster R-CNN [24] train val ResNet101 ResNet101 27.2 48.4 - - - - - - - - - -
SharpMask [39] train testdev ResNet50 VGG16 25.2 43.4 - - - - - - - - - -
Faster R-CNN [41] trainval testdev VGG16 VGG16 24.5 46.0 23.7 8.2 26.4 36.9 24.0 34.8 35.5 13.4 39.2 54.3
 [44] trainval testdev VGG16++ VGG16++ 27.5 49.2 27.8 8.9 29.5 41.5 25.5 37.4 38.3 14.6 42.5 57.4


Faster R-CNN trainval testdev VGG16 VGG16 23.3 44.7 21.5 9.4 27.1 32.0 22.7 36.8 39.4 18.3 44.0 56.2
Faster R-CNN trainval testdev ResNet101 ResNet101 31.5 52.8 33.3 13.6 35.4 44.5 28.0 43.6 45.8 22.7 51.2 64.1
Faster R-CNN trainval testdev IRNv2 IRNv2 34.7 55.5 36.7 13.5 38.1 52.0 29.8 46.2 48.9 23.2 54.3 70.8


TDM [ours] trainval testdev VGG16 + TDM VGG16 + TDM 28.6 48.1 30.4 14.2 31.8 36.9 26.2 42.2 44.2 23.7 48.3 59.3
TDM [ours] trainval testdev ResNet101 + TDM ResNet101 + TDM 35.2 55.3 38.1 16.6 38.4 47.9 30.4 47.8 50.3 27.8 54.9 67.6
TDM [ours] trainval testdev IRNv2 + TDM IRNv2 + TDM 37.3 57.8 39.8 17.1 40.3 52.1 31.6 49.3 51.9 28.1 56.6 71.1


Table 3: Object detection results on the COCO benchmark. Different methods use different networks for region proposal generation (ROINet) and for region classification (ClsNet). Results for the top block (except Faster R-CNN) were directly obtained from their respective publications [41, 24, 39, 44]. Faster R-CNN was reproduced by us. Middle block shows our implementation of Faster R-CNN framework, which we use as our primary baseline. Bottom block presents the main results of TDM network, with current state-of-the-art single-model performance highlighted.

To build a TDM network, we start with a standard bottom-up model trained on the detection task, and add progressively. The capacity for different , , and modules is given in Table 2. For the VGG16 network, we add top-down and lateral modules all the way to the conv1_x feature. Notice that the input feature dimension to RPN and RCN networks changes from 512 (for conv5_x) to 256 (for ), therefore we initialize the fc layers in RPN and RCN randomly [19]. However, since is same for the last three modules, we re-use the RPN and RCN layers for these modules.

For the ResNet101 and InceptionResNetv2, we add top-down and lateral modules for till conv1 and ‘Mixed_5b’ respectively. Similar to VGG16, their base networks are initialized with a model pre-trained on the detection task. However, as opposed to VGG16, where the RCN has just 2 fc layers, ResNet101 and InceptionResNetv2 models have an RCN with 9 and 48 convolutional layers respectively. This makes training RCN from random initialization difficult. To counter this, we ensure that all output feature dimensions () are same, so that we can be readily use pre-trained RPN and RCN. This is implemented using an additional convolutional layer wherever differs from (, all modules in ResNet101, and the final module in InceptionResNetv2).

We would like to highlight an issue with training of RPN at high-resolution feature maps. RPN is a fully convolutional module of Faster R-CNN, that generates an intermediate 512 dimension representation which is of the same resolution as input; and losses are computed at all pixel locations. This is efficient for coarse features (, last row in Table 1), but the training becomes prohibitively slow for finer resolution features. To counter this, we apply RPN at a stride which ensures that computation remains exactly the same (, using stride of 8 for in VGG16). Because of ‘ROI Pooling’ operations, RCN module still efficiently utilizes the finer resolution features.

5 Results

In this section, we evaluate our method on the task of object detection, and demonstrate consistent and significant improvement in performance when using features from the proposed TDM network.

Dataset and metrics.

All experiments and analysis in this paper are performed on the COCO dataset [33]. All models were trained on 40k train and 32k val images (which we refer to as ‘trainval’ set). All ablation evaluations were performed on 8k val images (‘minival

’ set) held out from the val set. We also report quantitative results on the standard testdev2015 split. For quantitative evaluation, we use the COCO evaluation metric of mean average precision (AP

111COCO [33] AP averages over classes, recall, and IoU levels.).

Experimental Setup.

We conduct experiments with three standard ConvNet architectures: VGG16 [46], ResNet101 [24] and InceptionResNetv2 [47]. All models (‘Baseline’ Faster R-CNN and ours) were trained with SGD for 1.5M mini-batch iterations, with batch size of 256 and 128 ROIs for RPN and RCN respectively. We start with an initial learning rate of 0.001 and decay it by 0.1 at 800k and 900k iterations.


Our primary baseline is using vanilla VGG16, ResNet101 and InceptionResNetv2 features in the Faster R-CNN framework. However, due to lack of implementations supporting all three ConvNets, we opted to re-implement Faster R-CNN in Tensorflow [1]. The baseline numbers reported in Table 3(middle) are using our implementation and training schedule and are generally higher than the ones reported in [41, 44]. Faster R-CNN was reproduced by us using the official implementation [41]. All other results in Table 3(top) were obtained from the original papers.

We also compare against models which use a single region proposal generator and a single region classifier network. In particular, we compare with SharpMask [39], because of its refinement modules with top-down and lateral connections, and [44] because they also augment the standard VGG16 network with top-down information. Note that different methods use different networks and train/test splits (see Table 3), making it difficult to do a comprehensive comparison. Therefore, for discussion, we will directly compare against our Faster R-CNN baseline (Table 3(middle)), and highlight that the improvements obtained by our approach are much bigger than the boosts by other methods.

5.1 COCO Results

In Table 3(bottom), we report results of the proposed TDM network on the testdev2015 split of the COCO dataset. We see that the TDM network leads to a 5.3 AP point boost over the vanilla VGG16 network ( AP AP), indicating that TDM learns much better features for object detection. Note that even though our algorithm is trained on less data (trainval), we outperform all methods with VGG16 architecture. For ResNet101, we improve the performance by 3.7 points to 35.2 AP. InceptionResNetv2 [47] architecture was the cornerstone of the winning entry to COCO 2016 detection challenge [26]. The best single model performance used by this entry achieves AP on the testdev split ([26]). Using InceptionResNetv2 as base, our TDM network achieves 37.3 AP, which is currently the state-of-the-art single model performance on the testdev split without any bells and whistles (, multi-scale, iterative box refinement, ). In fact, the TDM network outperforms the baselines (with same base networks) on almost all AP and AR metrics. Similarly, in Table 4, we observe that TDM achieves similar boosts across all network architectures on the minival split as well.

Figure 5: Improvement in AP over Faster R-CNN baseline. Base Networks: (left) VGG16, (middle) ResNet101, and (right) InceptionResNetv2. Improved performance for almost all categories emphasize the effectiveness of Top-Down Modulation for object detection. (best viewed digitally)

Figure 5 shows change in AP from Faster R-CNN baseline to the TDM network (for the testdev split). When using VGG16, for all but one category, TDM features improve the performance on object detection. In fact, more than 50% of the categories improve by 5 AP points or more, highlighting that the features are good for small and big objects alike. Similar trends hold for ResNet101 and InceptionResNetv2.

Improved localization.

In Table 3, we also notice that for VGG16, our method performs exceptionally well on the AP metric, improving the baseline Faster R-CNN by 8.9 AP points, which is much higher than the 3.5 point AP boost. We believe that using contextual features to select and integrate low-level finer details is they key reason for this improvement. Similarly for ResNet101 and InceptionResNetv2, we see 4.8 AP and 3.1 AP boost respectively.

Improvement for small objects.

In Table 3, for VGG16, ResNet101 and InceptionResNetv2), we see 4.8, 3 and 3.6 point boost respectively for small objects (AP) highlighting the effectiveness of features with TDM. Moreover, small objects are often on top of the list in Figure 5 (, sportsball +13 AP point, mouse +10 AP for VGG16). This is in line with other studies [20], which show that context is particularly helpful for some objects. Similar trends hold for the minival split as well: 5.6, 7.4 and 8.5 AP boost for VGG16, ResNet101 and InceptionResNetv2 respectively.


Method Net Features from: AP AP AP AP AP AP AR AR AR AR AR AR


Baseline VGG16 25.5 46.7 24.6 6.1 23.3 37.0 23.9 38.2 40.7 14.1 39.5 55.3
Skip-pool VGG16 25.3 46.3 25.9 9.1 24.0 36.0 24.6 40.0 42.4 18.6 41.8 54.1
TDM [ours] VGG16 + TDM 26.2 45.7 27.2 9.4 25.1 34.8 25.0 40.7 43.0 18.7 43.1 54.7
TDM [ours] VGG16 + TDM 28.8 48.6 30.7 11.0 27.1 37.3 26.5 42.7 45.0 21.1 44.2 56.4
TDM [ours] VGG16 + TDM 29.9 50.3 31.6 11.4 28.1 38.6 27.3 43.7 46.0 22.8 44.7 57.1
TDM [ours] VGG16 + TDM 29.8 49.9 31.7 11.7 28.0 39.3 27.1 43.5 45.9 23.9 45.4 56.8


Baseline ResNet101 32.1 53.2 33.8 9.4 29.7 45.7 28.3 44.3 46.7 19.3 46.3 60.9
TDM [ours] ResNet101 + TDM 34.4 54.4 37.1 10.9 31.8 48.2 30.1 47.5 49.8 21.7 49.1 64.0
TDM [ours] ResNet101 + TDM 35.3 55.1 38.3 11.2 33.0 48.2 30.7 48.0 50.5 22.5 50.1 63.6
TDM [ours] ResNet101 + TDM 35.7 56.0 38.5 16.8 39.2 49.0 30.9 48.5 50.9 28.1 55.6 68.5


Baseline IRNv2 35.7 56.5 38.0 8.9 32.0 52.5 30.8 47.8 50.3 19.6 49.9 66.9
TDM [ours] IRNv2 + TDM 37.3 57.9 39.5 11.4 33.3 53.3 32.8 49.1 51.5 22.7 50.6 67.5
TDM [ours] IRNv2 + TDM 38.1 58.6 40.7 17.4 41.1 54.7 32.4 50.1 52.6 28.9 57.2 72.3


Table 4: Ablation analysis on the COCO benchmark using the Faster R-CNN detection framework. All methods are trained on trainval and evaluated on minival set (Section 5). Methods are grouped based on their base network, best results are highlighted in each group.

Qualitative Results.

In Figure 6, we display qualitative results of the Top-down Modulation Network. Notice that the network can find small objects like remote (second row, last column; fourth row, first column) and sportsball (second row, third and fourth column). Also notice the detection results in the presence of heavy clutter.

Figure 6: Qualitative results of the proposed TDM network on randomly selected images from the minival set (best viewed digitally).

6 Design and Ablation Analysis

In this section, we perform control and ablative experiments to study the importance of top-down and lateral modules in the proposed TDM network.

6.1 How low should the Top-Down Modulation go?

In Section 4.3, we discussed the principles that we follow to add top-down and lateral modules. We connected these modules till the lowest layer choosing design decisions that hardware constraints would permit. However, is that overkill? Is there an optimal layer, after which this modulation does not help or starts hurting? To answer these questions, we limit the layer till which the modulation process happens, and use those features for Faster R-CNN.

Recall that the TDN network is built progressively, , we add one pair of lateral and top-down module at a time. So for this control study, we simply let each subsequent pair train for the entire learning schedule, treating the as the final output. We report the results on minival set in Table 4. As we can see, adding more top-down modulation helps in general. However, for VGG16, we see that the performance saturates at , and adding modules till do not seem to help much. Deciding the endpoint criteria for top-down modulation is an interesting future direction.

6.2 No lateral modules

To analyze the importance of lateral modules, and to control for the extra parameters added by the TDM network (Table 2), we train additional baselines with variants of VGG16 TDM network. In particular, we remove the lateral modules and use convolutional and upsampling operations from the top-down modules to train ‘deeper’ variants of VGG16 as baseline. To control for the extra parameters from lateral modules, we also increase the parameters in the convolutional layers. Note that for this baseline, we follow training methodology and design decisions used for training TDM networks.

As shown in Table 5(left), even though using more depth increases the performance slightly, the performance boost due to lateral modules is much higher. This highlights the importance of dividing the capacity of TDM network amongst lateral and top-down modules.

24.8 26.2
25.1 28.8
26.5 29.9
21.4 29.6


Pre-trained on:
COCO ImageNet
34.4 34.0
35.3 34.1


Table 5: (left) Importance of lateral modules: We use operations from the top-down module to increase depth of the VGG16 network; represents modified top-down module to account for more parameters. (right) Impact of Pre-training.

6.3 No top-down modules

Next we want to study the importance of the top-down path introduced by our TDM network. We believe that this path is responsible for transmitting contextual features and for selection of relevant finer details. Removing the top-down path exposes the ‘skip’-connections from bottom-up features, which can be used for object detection. We follow the strategy from [4], where they ROI-pool features from different layers, L2-normalize and concatenate these features and finally scale them back to the original conv5_3 magnitude and dimension.

We tried many variants of the Skip-pooling baseline, and report the best results in Table 4 (Skip-pool). We see that performance for small objects (AP) increases slightly, but overall the AP does not change much. This highlights the importance of using high-level contextual features in the top-down path for the selection of low-level features.

6.4 Impact of Pre-training

Finally, we study the impact of using a model pre-trained on the detection task to initialize our base networks and ResNet101/InceptionResNetv2’s RPN and RCN networks using only an image classification [8] pre-trained model. In Table 5(right), we see that initialization does not impact the performance by a huge margin. However, pre-training on the detection task is consistently better than using the classification initialization.

7 Conclusion

This paper introduces the Top-Down Modulation (TDM) network, which leverages top-down contextual features and lateral connections to bottom-up features for object detection. The TDM network uses top-down context to select low-level finer details, and learns to integrate them together. Through extensive experiments on the challenging COCO dataset, we demonstrate the effectiveness and importance of features from the TDM network. We show empirically that the proposed representation benefits all objects, big and small, and is helpful for accurate localization.

Even though we focused on the object detection, we believe these top-down modulated features will be helpful in a wide variety of computer vision tasks.