Detecting 11K Classes: Large Scale Object Detection without Fine-Grained Bounding Boxes

08/14/2019 ∙ by Hao Yang, et al. ∙ Amazon 2

Recent advances in deep learning greatly boost the performance of object detection. State-of-the-art methods such as Faster-RCNN, FPN and R-FCN have achieved high accuracy in challenging benchmark datasets. However, these methods require fully annotated object bounding boxes for training, which are incredibly hard to scale up due to the high annotation cost. Weakly-supervised methods, on the other hand, only require image-level labels for training, but the performance is far below their fully-supervised counterparts. In this paper, we propose a semi-supervised large scale fine-grained detection method, which only needs bounding box annotations of a smaller number of coarse-grained classes and image-level labels of large scale fine-grained classes, and can detect all classes at nearly fully-supervised accuracy. We achieve this by utilizing the correlations between coarse-grained and fine-grained classes with shared backbone, soft-attention based proposal re-ranking, and a dual-level memory module. Experiment results show that our methods can achieve close accuracy on object detection to state-of-the-art fully-supervised methods on two large scale datasets, ImageNet and OpenImages, with only a small fraction of fully annotated classes.



There are no comments yet.


page 8

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Object detection has been in the center stage of computer vision research, and the recent development of deep learning has vastly improved its performance. State-of-the-art algorithms, such as R-FCN 

[5], FPN [16] and Mask-RCNN [12] have achieved high accuracy on challenging benchmark datasets. However, these methods rely on accurate and complete annotations of object bounding boxes, which is expensive and time consuming to collect.

Exhaustively annotating bounding boxes for a large scale dataset with fine-grained level labels (here by large scale, we not only mean large number of instances , but also large number of categories ) is extremely laborious. Due to the multi-label nature of detection problem, the annotation costs is nearly . Even with state-of-the-art annotating methods [21, 22], it is still much more costly than annotating image level labels. Therefore, existing object detection datasets are either small scale and relative fully annotated, such as PASCAL VOC [8] and MS COCO [17], or large scale but only a part of the categories are annotated with bounding boxes, such as ImageNet [7] and OpenImages [14], and even for those categories with bounding boxes, missing labels are common [37].

Compared to annotating bounding boxes, annotating image level labels is cheaper. For example, ImageNet has K categories that are consider trainable (more than training images) with M total training images, but out of theses images, only K categories and below M training images have bounding boxes annotations [27] (close to M if we add all the ILSVRC-Detection annotations). Similarly, OpenImages has more than K trainable categories but only categories have bounding boxes [14].

Although researchers have developed several large scale algorithms, such as R-FCN 3000 [27] and SNIPER with soft sampling [37, 28] to facilitate detection on the scale of ImageNet and OpenImages detection set

. These methods still require full annotations of ground truth bounding boxes. This not only wastes a large part of the training data, which could potentially help learning a more robust detector, but also limits the detector’s capability to recognize more fine-grained categories. In this sense, fully-supervised detectors handicap our capability to utilize all training data and detect more categories. We believe a better way to train large-scale fine-grained detector is through semi-supervised learning with coarse-grained detection data and fine-grained classification data. There are several benefits for such semi-supervised detector. First, compared to fully-supervised detectors, the proposed semi-supervised detector only needs a relatively small amount of coarse-grained classes to be fully annotated. Intuitively, if we have fully annotated “dog”, we do not need bounding boxes for all the species of dogs. This greatly reduces the annotation cost and makes better use of existing data. Secondly, compared to weakly-supervised detectors, we can exploit the coarse-grained detection data and train a stronger detector. State-of-the-art weakly-supervised methods are still

points below fully-supervised methods [10, 39, 33] in mAP, while our semi-supervised method demonstrates that we can achieve comparable performance to fully-supervised methods.

The key to solve the problem, in our opinion, lies in the correlations between coarse-grained and fine-grained classes. Thus we are trying to answer the following two questions: 1) How to build proper correlations between fully-supervised coarse-grained data and weakly-supervised fine-grained data. 2) How to effectively utilize these correlations to transfer accurate object appearances learned from fully-supervised to weakly-supervised data, as well as to learn a better detector from the rich variances of more fine-grained weakly-supervised data.

To answer these questions, we propose an innovated large scale semi-supervised object detection solution. Compared to existing works [24, 27], our solution is able to handle both semantic and visual correlations between fully-supervised and weakly-supervised set, which makes the solution more extendable and applicable for more tasks. More importantly, beyond shared feature learning layers, we explicitly exploit these correlations to transfer knowledge from both data worlds. This not only helps us extend the detection to fine-grained level, but also improve the detector performance on coarse-grained labels with the help of diverse fine-grained data.

Specifically, we propose a two-stream network with a backbone based on R-FCN. One stream focuses on fully-supervised detection and the other stream solves the weakly-supervised detection with fine-grained data. These two streams share feature learning layers. Our main technical contributions include:

  • Fully-supervised detection stream augmentation with the fine-grained weakly-annotated data. With the shared backbone and multi-task training, we make use of more diverse fine-grained images with weak annotations to boost the detector performance.

  • Soft-attention based proposal re-ranking for weakly-supervised detection stream. We utilize the correlations between coarse- and fine-grained labels to bring attentions to proposals that are more likely to contain related objects from fully-supervised stream to the weakly-supervised stream.

  • Dual-level memory module with foreground attention pooling. We augment the network with an external memory module. Similar to clustering loss, this module can transfer knowledge from supervised to unsupervised data and further regularize the training process.

Results on two large scale datasets - OpenImages and ImageNet demonstrate that our method, without the need of bounding boxes annotations, can still achieve high detection accuracy on fine-grained classes, while also boosting the detection performance on coarse-grained classes in some cases. Furthermore, our designed framework is end-to-end trainable and almost as efficient as a standard detection network. The proposed components are well generalized and can be easily transferred to any 2-stage (RPN based) detectors in order to keep up with the rapid development of deep learning. The success of our method provides meaningful insights for practical detection data collection: to build a large scale object detector that is capable of detecting tens of thousands of classes (e.g. the 11K classes in ImageNet dataset), what we only need is to collect bounding boxes for a small set of coarse-grained classes and image labels for all fine-grained classes, which can significantly reduce the annotation cost.

2 Related Work

Fully-Supervised Object Detection: Fully-supervised detection can be divided into two categories: 1) One-stage methods, such as YOLO series [23, 24, 25] and SSD [18]. These methods do not require regional proposals and perform detection in a single shot. 2) Two-stage methods, such as Fast R-CNN series [11, 26], R-FCN [5], FPN [16] and Mask-RCNN [12]

. These methods build on the same idea that a detector should first generate regional proposals, which are likely to contain objects, then further classify the proposals into background and specific object classes. One-stage methods in general are faster in training and inference, but with lower detection accuracy, compared to two-stage methods. Our proposed framework is built on top of one of the state-of-the-art method, R-FCN with Deformable Convolutional Network 

[5, 6]. However, it can be easily adopted to any other two-stage detectors.

Weakly-Supervised Object Detection: Weakly-supervised object detection is often formulated as the key instance detection on multi-instance learning, where we view each object proposal as an instance and each image as a bag. The problem is to find out instances that contain objects, given only bag-level supervision. Most weakly-supervised detection methods have two stages: they first utilize Selective Search [36] or Edge Boxes [40] as proposals, and then use a CNN to solve the multi-instance learning problem. There are two main directions to further improve the performance of weakly supervised detection: improving the proposal quality and the aggregation-selection process of proposals. WSDDN [2] is one of the well-known work for weakly-supervised object detection with deep learning. The key idea is to have an additional ranking softmax for proposals to smartly aggregate and select proposal scores. OICR [30] improves WSDDN by incorporating multiple refinement streams with pseudo ground-truth.  [34] exploits web images to enhance training data.  [31] ditches hand-crafted object proposals in favor of a weakly-supervised version of Regional Proposal Network (RPN). Recently,  [10] also proposes to use extra attention map to improve the proposal selection process. In this work, we show that bounding box information of correlated coarse-grained detection classes can drastically improve the weakly-supervised fine-grained stream in both directions.

Semi-Supervised Object Detection: There are only a few research works in the semi-supervised detection field.  [32] proposes a LSDA-based method that can handle disjoint set semi-supervised detection, but this method is not end-to-end trainable and cannot be easily extended to state-of-the-art detection frameworks.  [35] proposes a semiMIL method on disjoint set semi-supervised detection, which achieves better performance than [32]. Note-RCNN [9] proposes a mining and training scheme for semi-supervised detection, but it needs seed boxes for all categories. YOLO 9000 [24] can also be viewed as a semi-supervised detection framework, but it is no more than a naive combination of detection and classification stream and only relies on the implicit shared feature learning from the network.

To the best of our knowledge, our method is the first semi-supervised fine-grained detection framework that explicitly exploit semantic/visual correlations between coarse-grained detection and fine-grained classification data. Experimental results on ImageNet and OpenImages datasets demonstrate that our setting is more applicable for real world large scale detection problems.

3 Technical Approach

In this section, we introduce how we solve the problem of semi-supervised fine-grained detection. We first formulate the problem in Section. 3.1, then introduce how we encode visual and semantic correlations in Section.3.2. Our overall architecture is outlined in Section.3.3. The key components are: a fully-supervised detection stream, a weakly-supervised stream with soft-attention based proposal re-ranking and a dual-level memory module with foreground attention pooling, which are explained in detail in Section.3.3.1, Section.3.3.2 and Section.3.3.3, respectively.

3.1 Problem Formulation

Let be the whole dataset consisting of a subset of classes with full ground-truth bounding box annotations ( denotes for fully-annotated), and a subset of classes with only image-level annotations ( denotes for weakly-annotated). Let and . Our goal is to train a detector that is able to accurately detect all classes. Our methods target to handle the challenging scenario, where the fully-annotated and weakly-annotated classes are disjoint (i.e., ), and there are much more image level annotations than bounding box annotations (i.e., ). This scenario includes the most prominent large-scale image datasets, namely ImageNet and OpenImages.

We assume that there exists semantic or/and visual correlations between fully-annotated set and weakly-annotated set . We also assume that contains all coarse-grained level labels and contains all fine-grained level labels to reflect real world scenarios. For example, for semantic correlation, we could have “dog” as a coarse-grained label with bounding boxes, and “labrador”, “chihuahua”, etc as fine-grained labels without bounding box annotations. Or for visual-semantic correlation, we could have “dog” as a coarse-grained label, and “wolf”, “coyote”, or even “stuffed dog” as fine-grained labels. Our method aims to utilize these correlations to provide accurate bounding boxes predictions for fine-grained classes and improve the training accuracy on coarse-grained classes with related fine-grained data.

3.2 Encoding Correlations between Coarse-Grained and Fine-Grained Classes

We believe the key of a successful semi-supervised detector is to build and utilize correlations between coarse-grained and fine-grained classes. These correlations are bridges that allow us to transfer knowledge between two worlds. Specifically, we consider two kinds of correlations: semantic and visual correlations.

3.2.1 Semantic Correlations

Semantic correlations are extracted from human knowledge and languages on how objects/concepts are structured and related. These correlations are often represented as a directed graph or tree, such as WordNet [20] and Visual Genome [15]

. The benefits of such semantic correlations are: 1) They encode strong prior knowledge on how we view the world. 2) They are readily available from various sources. For example, ImageNet is built on WordNet, and OpenImages is built on Google Knowledge Graph.

Encoding semantic correlations are straightforward as these correlations are already represented by a directed graph. For a coarse-grained detection class , if we consider its hyponyms as

, the encoding function can be written as a one-hot vector:


Here is an arbitrary fine-grained classification class. The reverse encoding from fine-grained to coarse-grained is just a similar function with hypernyms. We use these mapping functions in subsequent experiments.

There are drawbacks on semantic correlations. Firstly, adding new nodes to the existing graph requires a lot of expert efforts and is prune to errors. Secondly, semantic correlations are not always transferable to visual similarities, and vice versa. For example, “hyena” biologically is closer to “feline”, but visually more similar to “canine”, and “basketball” is visually similar to “orange”, but semantically they are far away. Therefore, we introduce visual correlations in the next section.

3.2.2 Visual Correlations

Visual correlations describe the visual similarities among objects/concepts. These correlations align better with training objectives, and are more flexible and easier to maintain. To encode visual correlations, we consider two scenarios, directly using detection classes as “super-classes” or building “super-classes” by clustering.

If we have relatively small number of coarse-grained classes, we could treat each of them as a “super-class” and build a two-level encoding. We obtain the -th object-class representation, , by taking the average of features

(extracted from the final layer of a deep neural network, such as ResNet-101, for sampled images

belonging to the -th class). After acquiring the representation for each class, we can then encode the correlations between coarse-grained and fine-grained classes through either hard-assignment or soft-assignment.

Let’s denote as the Euclidean distance between two representations from coarse-grained set and from fine-grained set, the hard-assignment encoding function is similar to Eq. 1:


Here is the class-specific threshold.

For soft-assignment, which is similar to weighted K-means clustering, we can assign a fine-grained class to multiple/all coarse-grained classes using a softmax function:


where is the temperature parameter that control the distribution of softmax function.

If we have a large number of coarse-grained classes, to reduce computational cost as well as facilitate more effective visual encoding, we can build a set of super-classes by (weighted, if we use soft assignment encoding) K-mean clustering on the representations of object classes from the coarse-grained set . We then have two encoding functions for and in the same spirit of Eq. 2 and Eq. 3.

3.3 Architecture

Figure 1: The overview of the designed architecture. The architecture can be split into three streams: 1) a fully-supervised detection stream, 2) a weakly-supervised classification stream, and 3) the Correlation components including the soft-attention based proposal re-ranking and a dual-level memory module. All three streams share common modules such as base CNN layers for feature learning and regional proposal network (RPN). During training, the detection data are used to train the RPN and R-FCN alike coarse-grained detection stream, while the fine-grained data are used to train the fine-grained classification stream. The correlation components are designed to transfer knowledge between coarse-grain and fine-grain.

To utilize the fully-supervised (coarse-grained) and weakly-supervised (fine-grained) data and their encoded correlations, we build three key components in our framework: 1) a fully-supervised detection stream for coarse-grained classes, 2) a weakly-supervised classification stream for fine-grained classes, and 3) correlation components to transfer knowledge between coarse-grained and fine-grained data,including shared backbone, soft-attention based proposal re-ranking, and the dual-level memory module. Details of these components are explained in the following sections.

3.3.1 Fully-Supervised Detection Stream

This stream is built on Deformable R-FCN [5]. First, a Regional Proposal Network (RPN) is used for generating proposals and is only trained with detection data to avoid label noises from weakly supervised data. Then, on the shared backbone, we apply position sensitive filters to pool features from each proposal. Since we have classes and filters per class, there are filters. After performing position-sensitive RoI pooling, we apply two fully connected layers to obtain final classification scores and regress bounding box results for each proposal. A cross-entropy loss and a bounding box regression loss are used for classification and regression learning, respectively:


where we use as the trade-off parameter [28].

3.3.2 Weakly-Supervised Detection Stream with Soft-Attention Proposal Re-ranking

Figure 2: The overview of weakly-supervised stream.

Weakly-supervised detection can be viewed as a special kind of multi-instance multi-label learning with key instance detection. Each image can be a bag in multi-instance learning and each proposal from the image is an instance in the bag. Since we just have the image(bag)-level labels, the key for solving this problem is how to aggregate proposal-level scores into image-level scores and how we select proposals that are most likely to contain target objects. In the context of deep learning and back-propagation, these two problems are closely tied together.

A classical way to aggregate proposal scores is to use max or average pooling [38]. This works well for getting good image-level predictions, but not for detecting proposals with the highest Intersection over Union (IoU) to ground truth bounding boxes. The most likely reason is that modern CNNs tend to focus on the most discriminative part of objects rather than the whole object [39]. For example, networks trained on ImageNet differentiate “person” from other objects by only using the head/face part rather than the whole body. Therefore, state-of-the-art weakly-supervised object detection methods [2, 34, 33, 10] employ some forms of regularization on the activation maps or the proposals selection to solve the problem.

Fortunately, since we have the fully-supervised detection stream, for each proposal, we actually know the presence or absence of closely related coarse-grained objects for a fine-grained object. Continuing with our “dog” example, considering if we are training the model for “chihuahua”, a proposal with high score on “dog” from the fully-supervised detection stream is much more likely to contain “chihuahua” than a proposal with low score on “dog”. This knowledge from the detection data is one good attention mechanism for weakly-supervised detection.

We design our weakly-supervised branch based on this idea. Similar to the fully-supervised stream, we use the shared RPN to generate proposals, and a RoI pooling layer to exact features for each proposal from shared layers. We select RoI pooling instead of PSRoI pooling to reduce the computation overhead, as PSRoI pooling requires times more filters than RoI pooling, and there are a large number of classes in our case. After we generate pooled features, we obtain proposal-level scores through a fully connected layer. These scores are then sent to two different pooling branches as shown in Fig.2

. The first branch is the global pooling: we use max, average or weighted average pooling to aggregate image-level scores, and normalized softmax loss to learn. This branch is designed for smoothing the training process and generating good image-level performance. The second branch is the attention pooling, which transfers knowledge from fully-supervised stream and helps create accurate proposals. Similar to soft-attention mechanism in neural machine translation 

[1], we use proposal scores from fully-supervised detection stream as an attention map and apply this map to the weakly-supervised detection scores to obtain final proposal scores. This score is then aggregated by average pooling.

Figure 3: Soft-attention based proposal re-ranking.

As shown in Fig.3, to obtain the attention map for ranking, we need to re-scale and normalize the coarse-grained detection scores, and then map the scores to fine-grained labels using previously discussed mapping functions, i.e., Eq. (1), (2), (3). Assuming we have a score map , where is the number of coarse-grained detection classes plus background, and is the number of proposals. If background class is at index , to constrain that each proposal on the map to one unique class, we use a softmax operation defined as:


where is the score for proposal at class . After obtaining , similar to the ranking term in WSDDN [2], we normalize the score map in the direction of proposals to obtain the coarse-grained attention map


However, since we need to apply the attention map to fine-grained proposals, we utilize the encoding function described in Section 3.2 as a coarse-to-fine mapping function to map the coarse-grained attention map to fine-grained. We obtain the fine-grained attention map at proposal as


where is the soft-assignment or one-hot hard-assignment encoding on class , with dimension . This attention map is then applied to the weakly-supervised score map by element-wise product.

The final loss function of the weakly-supervised fine-grained detection branch is:


where is a multi-label image level label and is the trade-off parameter which is set to be for all of our experiments. We use Top- average pooling for the first term and sum pooling for the second term. Detailed experiments on design choices can be found in our supplemental file.

3.3.3 Dual-Level Memory Module

Neural networks with memory are recently introduced to enable more powerful learning and reasoning ability for addressing several challenging tasks, such as question answering[19], one-shot learning[13] and semi-supervised classification [4]. Augmenting a network with an external memory component provides a similar role as the clustering loss in standard semi-supervised learning, but with dynamically updating feature representations and probabilistic predictions memorization.

Inspired by these works, we propose to add a memory module to our framework to take advantage of the memorable information generated in model learning and to further regularize the learning. Unlike [4], our semi-supervised fine-grained detection task is a two-level semi-supervised problem. For fully-supervised detection stream, we do not have proposal-level coarse-grained labels for the fine-grained classification data, and for weakly-supervised stream, we do not have image-level fine-grained labels for the coarse-grained detection data. Therefore, we need two levels of memory, coarse-grained proposal-level memory and fine-grained image-level memory. With these principles in mind, we propose a Dual-Level Memory module with Foreground Attention pooling (DLM-FA).

For the coarse-grained level memory, we have proposal(box)-level labels for coarse-grained detection images, but we are lacking proposal-level coarse-grained labels for fine-grained images. If we view each proposal as one training instance, we are facing a straightforward semi-supervised learning problem and we can directly use the same memory structure as in [4]. The loss function is:


where is the entropy and is the Kullback-Leibler (KL) divergence. is the memory prediction and is the network prediction for each proposal .

For the fine-grained level memory, we have image-level labels for fine-grained classification images, but we are lacking fine-grained labels for coarse-grained images. However, compared with standard semi-supervised setting, we are dealing with multi-instance semi-supervised scenario. Therefore, we need to aggregate proposal-level features and predictions to image-level in order to facilitate memory update and prediction. We use Foreground Attention (FA) pooling to filter out noisy proposals. In FA pooling, we only pool features and predictions from proposals with high responses of positive image-level, and aggregate these features and predictions by sum pooling to represent their corresponding images. Specifically, if image has proposals , and their corresponding features and scores are and , the image-level feature for a specific class is then defined by:


The image-level predictions is pooled in a similar way. After we have the image-level features and scores to update memory modules, we utilize


as the loss function for fine-grained level memory. The whole memory loss is then:


Details of the memory module can be found in the supplementary file.

4 Experiments

In this section, we first introduce our implementation details. Then we discuss the experimental results and compare them with other baseline methods. We test our method on two most challenging large scale datasets – OpenImages [14] and ImageNet [7].

4.1 Implementation Details and Baselines

Our implementation is based on SNIPER [28]. In particular, we use mixed precision training for larger batch size and faster training speed. ResNet- with fp weights is used as the shared backbone. We use fp weights for the fully connected and convolutional layers of all heads. We train the model on V- GPUs with a batch size of (i.e. per GPU). A balance sampling scheme is used for detection and classification data, i.e., we sample the same number of classification and detection data for each batch. The initial learning rate is set to be for all experiments. We train all models for epochs and the learning rate is dropped by for every epochs. Image horizontal flipping is used for data augmentation. We only use one scale, namely for both training and testing. During inference, we run soft NMS [3]

on the model outputs with the standard deviation parameter of

in the Gaussian weighting function.

We compare our method with state-of-the-art fully-supervised detection methods trained on the same data. All results are reported in mean average precision (mAP) with intersection-over-union (IoU) threshold at . Specifically, our method is compared to: 1) SNIPER-CG-Fully: SNIPER trained on coarse-grained data with fully annotated bounding boxes. 2) SNIPER-FG-Fully, i.e., SNIPER trained on fine-grained data with fully annotated bounding boxes. 3) SNIPER-FG-Weakly: SNIPER with fixed backbone and RPN trained from coarse-grained detection data and fine-tune on fine-grained data with only image-level labels. This is one of the strongest weakly baselines in our test. 4) SNIPER-All: SNIPER trained on coarse- and fine- grained data with all bound box and label annotations. 111We have also compared with semi-supervised detection methods  [29, 32, 35] in same-granularity random-split setting. Since this is not the main focus of our paper and due to space limitations, the results are shown in supplemental file.

Figure 4: Qualitative Results. The first row shows the fine-grained detection results from our best OpenImages model (last row of Table 2). The second row shows fine-grained detection results from our ImageNet model on fine-grained classes.

4.2 OpenImages Results

Class classes training test
Coarse-grained 34 786K 22K
Fine-grained 462 567K 8.8K
Table 1: Statistics for OpenImages dataset.

OpenImagesV4 dataset contains bounding box annotations for classes with a semantic tree based on Google Knowledge Graph. We use all the leaf nodes in the semantic tree as fine-grained classes. These fine-grained classes have direct parent-nodes that can be used as coarse-grained classes. However, since these classes also have hierarchy, for simplicity, we merged the lower level classes into their highest level parent classes. We end up with coarse-grained classes. As discussed in the previous section, we only use bounding box annotations of the coarse-grained classes, and image-level labels for fine-grained classes in training. We evaluate our model on the OpenImages validation dataset for object detection. The validation dataset contains bounding box annotations for all coarse-grained and fine-grained labels. Table 1 shows the statistics for the coarse-grained and fine-grained data in training and validation datasets of OpenImages.

Results for different models are sumarrized in Table 2. We can see that naive combination of the fully-supervised and weakly-supervised stream without soft-attention and memory can already perform pretty well on both tasks, especially on coarse-grained detection. The naive baseline already outperforms SNIPER-CG-Fully by around points. This implies that the shared backbone benefits a lot from the rich variety of the closely related fine-grained images, as well as the multi-task training. However, due to the simple proposal aggregation method (top-5 pooling), we can see that the fine-grained detection results, though reasonable, are far lower than SNIPER-FG-Fully and SNIPER-All. We can also see that naive weakly-supervised stream still outperforms the pure weakly-supervised method on the same RPN, which further validates the benefits of joint training and shared backbone.

If we add soft-attention based proposal re-ranking during training and inference, the on fine-grained class increases by over points. It is clear shown that, bringing the knowledge learned from the detector in can significantly help the weakly-supervised stream. For the memory module, if we just add a single-level coarse-grained memory module similar to [4], we do not see any improvement. However, with the help of Dual-Level Memory module, we can further minimize the gap between our fine-grained stream and the fully-supervised one.

Method Training Data/Label mAP-CG mAP-FG
SNIPER [28] CG-Fully 45.7 -
SNIPER [28] FG-Fully - 59.1
SNIPER [28] All 28.7 54.0
SNIPER-Weakly FG-Weakly - 20.2
Naive CG-Fully+FG-Weakly 52.5 34.0
+Soft-Attention CG-Fully+FG-Weakly 52.9 49.2
+CG-Memory CG-Fully+FG-Weakly 52.2 49.6
+DLM-FA CG-Fully+FG-Weakly 53.5 51.9
Table 2: Test results for different models on OpenImages dataset where we evaluated coarse-grained (CG) and fine-grained (FG) classes separately.

Although it shows that our best model is still points lower than the best fully-supervised model on fine-grained classes, we should mention that, our model actually performs similarly with SNIPER-FG-Fully in terms of , both at around . This demonstrates that though we do not have bounding boxes on fine-grained classes, we can still learn to predict as accurately as fully-supervised methods.

4.3 ImageNet Results

Class classes training test
Coarse-grained 200 400K 22K
Fine-grained-3K 2937 870K 46K
Fine-grained-11K 11021 1.7M -
Table 3: Statistics for ImageNet dataset.
Method Training Data/Label mAP-CG mAP-FG
SNIPER [28] CG-Fully 54.0 -
SNIPER [28] k-FG-Fully - 41.6
YOLO- [24] COCO+-FG-Weakly 19.9 -
R-FCN- [27] k-FG-Fully 34.9 -
Ours-3K CG-Fully+k-FG-Weakly 50.7 35.1
Ours-11K CG-Fully+k-FG-Weakly 49.1 -
Table 4: Test results for different models on ImageNet dataset where we evaluated coarse-grained (CG) and fine-grained (FG) classes separately. Please note that YOLO- and R-FCN- results are not directly comparable. Based on [27], SNIPER--FG-Fully should be the performance upper bound of R-FCN- on FG.

We then run experiments on ImageNet dataset. As shown in Table 3, we use the ILSVRC 2014 Detection set with classes as the coarse-grained set. Two fine-grained sets are tested. One is the set with bounding box annotations, similar to what is used in [27]. For this set, we split of the training data to the validation set to test the performance of our fine-grained stream. The other set contains K classes with each class having over training images. There are in total 13M images in this set. In our experiment, we randomly sample images from all classes for training and testing. Such a subset could have been general enough and is a good representative of the full set. We build visual correlations between the fine-grained and coarse-grained classes using the soft assignment introduced before.

From the results summarized in Table 4, we can see that unlike results from OpenImages dataset, our coarse-grained detection result is slightly worse than SNIPER-CG-Fully, by margins of and , respectively. This could be explained by the correlations of coarse-grained and fine-grained data. For OpenImages, the coarse-grained and fine-grained sets are much closely correlated as they are hand-picked to form a compact semantic tree, while for ImageNet, the fine-grained sets are arbitrary picked by the availability of bounding box annotations and the number of training images. The ImageNet dataset could contain much more diverse classes in terms of semantic and visual correlations between the coarse-grained set. Therefore, we do not observe improvement on the coarse-grained performance.

Similar to OpenImages, though our best model is points worse than the best supervised model on fine-grained classes in , our model actually performs better than SNIPER-FG-Fully in terms of , with our model at and SNIPER-FG-Fully at . This again demonstrates that we can learn to predict as accurately as fully-supervised methods on large scale datasets.

We also present the detection results of YOLO- and R-FCN- on the CG validation set in Table 4. Note that these methods are not trained on the same data as our method, thus the results cannot be directly compared. However, what we would like to present here is that we are able to train a large-scale detector that is capable of detecting up to k classes while still outperforming YOLO- and R-FCN- by a large margin on coarse-grained classes.

Since there is no bounding box annotations for the K classes in ImageNet, we demonstrate the qualitative results in Figure 4. Overall, we find our model perform reasonably well. For example, our model is able to detect fine-grained animal species and human activities. More qualitative results and failure cases can be found in the supplemental file.

5 Conclusion

In this paper, we have proposed a semi-supervised based method to solve large-scale fine-grained object detection problem. Our method can achieve comparable results as state-of-the-art fully-supervised detector, by utilizing data from only a small number of fully-annotated coarse-grained classes and large scale weakly-annotated fine-grained classes. Our work not only establishes a new way of learning large scale detector, but also provides insights for large scale data collection and annotation.

There are a few future directions to explore. Currently we just use a simple two-level tree structure, and have not explored correlations within the coarse-grained or fine-grained set. Apparently, k fine-grained classes should not be treated as flat and uncorrelated. we should consider utilized the hierarchical structures within the fine-grained classes. We could also try to explore systemic ways of find good fine-grained classes that can be reasonable well detected and also helpful for coarse-grained detection.


  • [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014.
  • [2] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In CVPR, pages 2846–2854, 2016.
  • [3] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms - improving object detection with one line of code. In ICCV, pages 5562–5570, 2017.
  • [4] Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In ECCV, 2018.
  • [5] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: object detection via region-based fully convolutional networks. In NIPS, pages 379–387, 2016.
  • [6] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, pages 764–773, 2017.
  • [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009.
  • [8] Mark Everingham, Luc J. Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303–338, 2010.
  • [9] Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia. NOTE-RCNN: noise tolerant ensemble RCNN for semi-supervised object detection. CoRR, abs/1812.00124, 2018.
  • [10] Weifeng Ge, Sibei Yang, and Yizhou Yu. Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning. In CVPR, June 2018.
  • [11] Ross B. Girshick. Fast R-CNN. In ICCV, pages 1440–1448, 2015.
  • [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pages 2980–2988, 2017.
  • [13] Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. CoRR, abs/1703.03129, 2017.
  • [14] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from, 2017.
  • [15] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016.
  • [16] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, pages 936–944, 2017.
  • [17] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740–755, 2014.
  • [18] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. In ECCV, pages 21–37, 2016.
  • [19] Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP, pages 1400–1409, 2016.
  • [20] George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, 1995.
  • [21] Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vittorio Ferrari. We don’t need no bounding-boxes: Training object class detectors using only human verification. In CVPR, pages 854–863, 2016.
  • [22] Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vittorio Ferrari. Extreme clicking for efficient object annotation. In ICCV, pages 4940–4949, 2017.
  • [23] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pages 779–788, 2016.
  • [24] Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In CVPR, pages 6517–6525, 2017.
  • [25] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/1804.02767, 2018.
  • [26] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE T-PAMI, 39(6):1137–1149, 2017.
  • [27] Bharat Singh, Hengduo Li, Abhishek Sharma, and Larry S. Davis. R-FCN-3000 at 30fps: Decoupling detection and classification. 2018.
  • [28] Bharat Singh, Mahyar Najibi, and Larry S Davis. SNIPER: Efficient multi-scale training. In NIPS, 2018.
  • [29] Krishna Kumar Singh, Santosh Kumar Divvala, Ali Farhadi, and Yong Jae Lee. DOCK: detecting objects by transferring common-sense knowledge. In ECCV, pages 506–522, 2018.
  • [30] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. CoRR, abs/1704.00138, 2017.
  • [31] Peng Tang, Xinggang Wang, Angtian Wang, Yongluan Yan, Wenyu Liu, Junzhou Huang, and Alan L. Yuille. Weakly supervised region proposal network and object detection. In ECCV, pages 370–386, 2018.
  • [32] Yuxing Tang, Josiah Wang, Boyang Gao, Emmanuel Dellandréa, Robert J. Gaizauskas, and Liming Chen. Large scale semi-supervised object detection using visual and semantic knowledge transfer. In CVPR, pages 2119–2128, 2016.
  • [33] Qingyi Tao, Hao Yang, and Jianfei Cai. Exploiting web images for weakly supervised object detection. IEEE TMM, 2018.
  • [34] Qingyi Tao, Hao Yang, and Jianfei Cai. Zero-annotation object detection with web knowledge transfer. In ECCV, 2018.
  • [35] Jasper R. R. Uijlings, Stefan Popov, and Vittorio Ferrari. Revisiting knowledge transfer for training object class detectors. In CVPR, pages 1101–1110, 2018.
  • [36] Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, and Arnold W. M. Smeulders. Selective search for object recognition. IJCV, 104(2):154–171, 2013.
  • [37] Zhe Wu, Navaneeth Bodla, Bharat Singh, Mahyar Najibi, Rama Chellappa, and Larry S. Davis. Soft sampling for robust object detection. CoRR, abs/1806.06986, 2018.
  • [38] Hao Yang, Joey Tianyi Zhou, Jianfei Cai, and Yew-Soon Ong. MIML-FCN+: multi-instance multi-label learning via fully convolutional networks with privileged information. In CVPR, pages 5996–6004, 2017.
  • [39] Yongqiang Zhang, Yancheng Bai, Mingli Ding, Yongqiang Li, and Bernard Ghanem. W2f: A weakly-supervised to fully-supervised framework for object detection. In CVPR, June 2018.
  • [40] C. Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, pages 391–405, 2014.