Given an image, an object detection/localization method aims to recognize and locate objects of interest within it. It is one of the most widely studied problems in computer vision with a variety of applications. Most object detectors adopt strong supervision in learning appearance models of object categories, that is by using training images annotated with bounding boxes encompassing the objects of interest, along with their category labels. The recent success of deep convolutional neural networks (CNN) for object detection, such as DetectorNet , OverFeat , R-CNN , SPP-net , Fast R-CNN , Faster R-CNN , YOLO  and SSD , is heavily dependent on a large amount of training data manually labeled with object localizations ( _meaning:NTF . e.g _catcode:NTF a e.g. e.g., PASCAL VOC 
, ILSVRC (subset of ImageNet), and Microsoft COCO  datasets).
Although localized object annotations are extremely valuable, the process of manually annotating object bounding boxes is extremely laborious and unreliable, especially for large-scale databases. On the other hand, it is usually much easier to obtain annotations at image level ( _meaning:NTF . e.g _catcode:NTF a e.g. e.g., from user-generated tags on Flickr or Web queries). For example, ILSVRC contains image-level annotations for 1,000 categories, while object-level annotations are currently restricted to only 200 categories. One could apply image-level classifiers directly to detect object categories, but this will result in a poor performance as there are differences in the statistical distribution between the training data (whole images) and the test data (localized object instances). Previous work by Hoffman _meaning:NTF . et al _catcode:NTF a et al. et al. addresses this issue, by learning a transformation between CNN classifiers and detectors of object categories with both image-level and object-level annotations (“strong” categories), and applying the transformation to adapt image-level classifiers to object detectors for categories with only image-level labels (“weak” categories). Part of this work involves transferring category-specific classifier and detector differences of visually similar “strong” categories equally to a classifier of a “weak” category to form a detector for that category (Fig. 1). We argue that more can potentially be exploited from such similarities in an informed manner to improve detection beyond using the measures solely for nearest neighbor selection (see Section 4.1). Moreover, since there is evidence that deep CNNs trained for image classification also learn proxies to objects and object parts , the transformation from CNN classifiers to detectors is reasonable and practicable.
Our main contribution in this paper is therefore to incorporate external knowledge about object similarities from visual and semantic domains in modeling the aforementioned category-specific differences, and subsequently transferring this knowledge for adapting an image classifier to an object detector for a “weak” category. Our proposed method is motivated by the following observations: (i) category specific difference exists between a classifier and a detector [4, 13]; (ii) visually and semantically similar categories may exhibit more common transferable properties than visually or semantically dissimilar categories; (iii) visual similarity and semantic relatedness are shown to be correlated, especially when measured against object instances cropped out from images (thus discarding background clutter) . Intuitively, we would prefer to adapt a cat classifier to a cat detector by using the category-specific differences between the classifier and the detector of a dog rather than of a violin or a strawberry (Fig. 2). The main advantage of our proposed method is that knowledge about object similarities can be obtained without requiring further object-level annotations, for example from existing image databases, text corpora and external knowledge bases.
Our work aims to answer the question: can knowledge about visual and semantic similarities of object categories (and the combination of both) help improve the performance of detectors trained in a weakly supervised setting ( _meaning:NTF . i.e _catcode:NTF a i.e. i.e., by converting an image classifier into an object detector for categories with only image-level annotations)? Our claim is that by exploiting knowledge about objects that are visually and semantically similar, we can better model the category-specific differences between an image classifier and an object detector and hence improve detection performance, without requiring bounding box annotations. We also hypothesize that the combination of both visual and semantic similarities can help further improve the detector performance. Experimental results on the challenging ILSVRC2013 dataset  validate these claims, showing the effectiveness of our approach of transferring knowledge about object similarities from both visual and semantic domains to adapt image classifiers into object detectors in a semi-supervised manner.
A preliminary version of this work appeared in . In this paper, we provide more technical details of our models, introduce a bounding-box post-processing method based on the transferability of regression models, and present extended results with more comparisons. The rest of the paper is organized as follows. We review related work in Section 2 and define the semi-supervised object detection problem in Section 3. In Section 4, we first review the LSDA framework, and then introduce our two knowledge transferring methods ( _meaning:NTF . i.e _catcode:NTF a i.e. i.e. visual similarity based method and semantic similarity based method). We present our experimental results and comparisons in Section 5. In Section 6, we conclude and describe future direction.
2 Related Work
With the remarkable success of deep CNN on large-scale object recognition  in recent years, a substantial number of CNN-based object detection frameworks [2, 3, 4, 6, 5, 7, 8, 9] have emerged. However, these object detectors are trained in a fully supervised manner, where bounding box annotations are necessary during training. The requirement of bounding box annotations hinders the application of these methods in large-scale datasets where training images are weakly annotated.
2.1 Weakly Supervised Learning
Weakly supervised learning methods for object detection attempt to learn localization cues from image-wide labels indicating the presence or absence of object instances of a category, thus reducing or removing the requirement of bounding box annotations [17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. Recently, there have been several studies in CNN-based object detection in a weakly-supervised setting [27, 28, 29, 30, 31], _meaning:NTF . i.e _catcode:NTF a i.e. i.e. using training images with only image-level labels and no bounding boxes. The common practice is to jointly learn an appearance model together with the latent object location from such weak annotations. Such approaches only adopt CNN as a feature extractor, and exhaustively mine image regions extracted by region proposal approaches, such as Selective Search , BING , and EdgeBoxes . Oquab _meaning:NTF . et al _catcode:NTF a et al. et al. develop a weakly supervised CNN end-to-end learning pipeline that learns from complex cluttered scenes containing multiple objects by explicitly searching over possible object locations and scales in the image, which can predict image labels and coarse locations (but not exact bounding boxes) of objects. Bilen and Vedaldi  propose a Weakly Supervised Deep Detection Network (WSDNN) method that extends a pre-trained network to a two-stream CNN: recognition and detection. The recognition and detection scores for region proposals are aggregated to predict the object category. Zhou _meaning:NTF . et al _catcode:NTF a et al. et al. adopt a classification-trained CNN to learn to localize object by generating Class Activation Maps (CAM) using global average pooling (GAP). Hoffman _meaning:NTF . et al _catcode:NTF a et al. et al. propose a Large Scale Detection through Adaptation (LSDA) algorithm that learns the difference between the CNN parameters of the image classifier and object detector of a “fully labeled” category, and transfers this knowledge to CNN classifiers for categories without bounding box annotated data, turning them into detectors. For LSDA, auxiliary object-level annotations for a subset of the categories are required for training “strong” detectors. This can be considered a semi-supervised learning problem (see Section 3). We improve upon LSDA, by incorporating knowledge about visual and semantic similarities of object categories during the transfer from a classifier to a detector.
2.2 Transfer Learning
Another line of related work is to exploit knowledge transfer from various domains. Transfer learning (TL)  aims to transfer knowledge across different domains or tasks. Two general categories of TL have been proposed in previous work: homogeneous TL [39, 40, 13] in a single domain but with different data distributions in training and testing sets, and heterogeneous TL [41, 42, 43] across different domains or modalities. LSDA treats the transfer from classifiers to detectors as a homogeneous TL problem as the data distributions for image classification (whole image features) and object detection (image region features) are different. The adaptation from a classifier to a detector is, however, restricted to the visual domain. Lu _meaning:NTF . et al _catcode:NTF a et al. et al. propose a sparse representation based discriminative knowledge transfer method that leverages relatedness of various source categories with the target category, where only few training samples existed, to enhance learning of the target classifier. Rochan and Wang  propose an appearance transfer method by transferring semantic knowledge (heterogeneous TL) from familiar objects to help localize novel objects in images and videos. Singh _meaning:NTF . et al _catcode:NTF a et al. et al. transfer tracked object boxes from weakly labeled videos to weakly labeled images to automatically generate pseudo ground-truth bounding boxes. Our work integrates knowledge transfer via both visual similarity (homogeneous TL) and semantic relatedness (heterogeneous TL) to help convert classifiers into detectors. Frome _meaning:NTF . et al _catcode:NTF a et al. et al. present a deep visual-semantic embedding model learned to recognize visual objects using both labeled image data and semantic information collected from unannotated text. Shu _meaning:NTF . et al _catcode:NTF a et al. et al. propose a weakly-shared Deep Transfer Network (DTN) that hierarchically learns to transfer semantic knowledge from web texts to images for image classification, building upon Stacked Auto-Encoders . DTN takes auxiliary text annotations (user tags and comments) and image pairs as input, while our semantic transfer method only requires image-level labels.
2.3 Semantic Similarity of Text
Semantic similarity is a well-explored area within the Natural Language Processing community. The main objective is to measure the distance between thesemantic meanings of a pair of words, phrases, sentences, or documents. For example, the word “car” is more similar to “bus” than it is to “cat”. The two main approaches to measuring semantic similarity are knowledge-based approaches and corpus-based, distributional methods. In the case of knowledge-based approaches, external resources such as thesauri (primarily WordNet ) or online knowledge bases are used to compute the similarity between the semantic meaning of two terms, for example using path-based similarity  or information-content similarity measures [50, 51]. The heavy reliance on knowledge bases, which tend to suffer from issues such as missing words, resulted in the development of distributional methods that rely instead on text corpora. In such methods, each term is represented as a contextvector, and two terms are assumed to have similar vectors if they occur frequently in the same context (e.g. “car” and “truck” have similar vectors because they often co-occur with “drive” and “road”). Such context vectors are more often referred to in recent years as word embeddings. Recent advances in word embeddings trained on large-scale text corpora [52, 53] have helped progress research in distributional methods to semantic similarity, as it has been observed that semantically related word vectors tend to be close in the embedding space, and that the embeddings capture various linguistic regularities  (King - Man + Woman Queen). As such, we will concentrate on such state-of-the-art word embedding methods to measure the semantic similarity of terms.
3 Task Definition
In our semi-supervised learning case, we assume that we have a set of “fully labeled” categories and “weakly labeled” categories. For the “fully labeled” categories, a large number of training images with both image-level labels and bounding box annotations are available for learning the object detectors. For each of the “weakly labeled” categories, we have many training images containing the target object, but we do not have access to the exact locations of the objects. This is different from the semi-supervised learning proposed in previous work [55, 56, 57], where typically a small amount of fully labeled data with a large amount of weakly labeled (or unlabeled) data are provided for each category. In our semi-supervised object detection scenario, the objective is to transfer the trained image classifiers into object detectors on the “weakly labeled” categories.
4 Similarity-based Knowledge Transfer
We first describe the Large Scale Detection through Adaptation (LSDA) framework , upon which our proposed approach is based (Section 4.1). We then describe our proposed knowledge transfer models with the aim of improving LSDA. Two knowledge domains are explored: (i) visual similarity (Section 4.2); (ii) semantic relatedness (Section 4.3). Next, we combine both models to obtain our mixture transfer model, as presented in Section 4.4. Finally, we propose to transfer the knowledge to bounding-box regression from fully labeled categories to weakly labeled categories in Section 4.5.
4.1 Background on LSDA
Let be the dataset of categories to be detected. One has access to both image-level and bounding box annotations only for a set of () “fully labeled” categories, denoted as , but only image-level annotations for the rest of the categories, namely “weakly labeled” categories, denoted as . Hence, a set of image classifiers can be trained on the whole dataset (), but only object detectors (from ) can be learned according to the availability of bounding box annotations. The LSDA algorithm learns to convert image classifiers (from ) into their corresponding object detectors through the following steps:
Pre-training: First, an 8-layer (5 convolutional layers and 3 fully-connected () layers) Alex-Net  CNN is pre-trained on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 classification dataset , which contains 1.2 million images of 1,000 categories.
Fine-tuning for classification: The final weight layer (1,000 linear classifiers) of the pre-trained CNN is then replaced with linear classifiers. This weight layer is randomly initialized and the whole CNN is then fine-tuned on the dataset . This produces a classification network that can classify categories (i.e., -way softmax classifier), given an image or an image region as input.
Category-invariant adaptation: Next, the classification network is fine-tuned into a detector with bounding boxes of as input, using the R-CNN  framework. As in R-CNN, a background class () is added to the output layer and fine-tuned using bounding boxes from a region proposal algorithm, _meaning:NTF . e.g _catcode:NTF a e.g. e.g., Selective Search . The layer parameters are category specific, with 4,097 weights ( output: 4,096, plus a bias term) in each category, while the parameters of layers 1-7 are category invariant. Note that object detectors are not able to be directly trained on , since the fine-tuning and training process requires bounding box annotations. Therefore, at this point, the category specific output layer stays unchanged. The variation matrix of after fine-tuning is denoted as .
Category-specific adaptation: Finally, each classifier of categories
is adapted into a corresponding detector by learning a category-specific transformation of the model parameters. This is based on the assumption that the difference between classification and detection of a target object category has a positive correlation with those of similar (close) categories. The transformation is computed by adding a bias vector to the weights of. This bias vector for category is measured by the average weight change of its nearest neighbor categories in set , from classification to detection.
where is the weight variation of the nearest neighbor category in set for category . and are, respectively, layer weights for the fine-tuned classification and the adapted detection network. The nearest neighbor categories are defined as those with nearest -norm (Euclidean distance) of weights in set .
The fully adapted network is able to detect all categories in test images. In contrast to R-CNN, which trains SVM classifiers on the output of the layer followed by bounding box regression on the extracted features from the layer of all region proposals, LSDA directly outputs the score of the softmax “detector”, and subtracts the background score from this as the final score. This results in a small drop in performance, but enables direct adaptation from a classification network into a detection network on the “weakly labeled” categories, and significantly reduces the training time.
Hoffman _meaning:NTF . et al _catcode:NTF a et al. et al. demonstrated that the adapted model yielded a 50% relative mAP (mean average precision) boost for detection over the classification-only framework on the “weakly labeled” categories of the ILSVRC2013 detection dataset (from 10.31% to 16.15%). They also showed that category-specific adaptation (final LSDA step) contributes least to the performance improvement (16.15% with vs. 15.85% without this step), with the other features (adapted layers 1-7 and background class) being more important. However, we found that by properly adapting this layer, a significant boost in performance can be achieved: an mAP of 22.03% can be obtained by replacing the semi-supervised weights with their corresponding supervised network weights and leaving the other parameters fixed. Thus, we believe that adapting this layer in an informed manner, such as making better use of knowledge about object similarities, will help improve detection.
In the next subsections, we will introduce our knowledge transfer methods using two different kinds of similarity measurements to select the nearest categories and weight them accordingly to better adapt the layer, which can efficiently convert an image classifier into an object detector for a “weakly labeled” category.
4.2 Knowledge Transfer via Visual Similarity
Intuitively, the object detector of an object category may be more similar to those of visually similar categories than of visually distinct categories. For example, a cat detector may approximate a dog detector better than a strawberry detector, since cat and dog are both mammals sharing common attributes in terms of shape (both have four legs, two ears, two eyes, one tail) and texture (both have fur). Therefore, given a “fully labeled” dataset and a “weakly labeled” dataset , our objective is to model the visual similarity between each category and all the other categories in , and to transfer this knowledge for transforming classifiers into detectors for .
Visual similarity measure: Visual similarity measurements are often obtained by computing the distance between feature distributions such as the or output of a CNN, or in the case of LSDA the layer parameters. In our work, we instead forward propagate an image through the whole fine-tuned classification network (created by the second step in Section 4.1) to obtain a
-dimensional classification score vector. This score vector encodes the probabilities of an image being each of theobject categories. Consequently, for all the positive images of an object category , we can directly accumulate the scores of each dimension, on a balanced validation dataset. We assume that the normalized accumulated scores (range [0,1]) imply the similarities between category and other categories: the larger the score, the more it visually resembles category . This assumption is supported by the analysis of deep CNNs [58, 59, 60]: CNNs are apt to confuse visually similar categories, on which they might have higher prediction scores. The visual similarity (denoted ) between a “weakly labeled” category and a “fully labeled” category is defined as:
where is a positive image from category of the validation set of , is the number of positive images for this category, and is the
CNN output of the softmax layer on, namely, the probability of being category as predicted by the fine-tuned classification network. is the degree of similarity after normalization on all the categories in .
Note that we adopt the outputs (classification scores) since most of the computation is integrated into the end-to-end Alex-Net framework except for the accumulation of classification scores in the end, saving the extra effort otherwise required for distance computation if or outputs were to be used. The idea of using the distance of the weights (linear classifier parameters) as a visual similarity measurement in LSDA is closely related to ours. However, in addition to the weights, our visual similarity measurement is assumed to leverage the powerful and supplementary feature representations generated by the prior layers of the neural network by combining both, given the fact that the outputs are obtained by taking the inner-product of outputs (visual features) and weights. Experimental results in Section 5 validate this intuition.
Weighted nearest neighbor scheme: Using Eq. (1), we can transfer the model parameters based on a category’s nearest neighbor categories selected by Eq. (2). This allows us to directly compare our visual similarity measure to that of LSDA which uses the Euclidean distance between the parameters. An alternative to Eq. (1) is to consider a weighted nearest neighbor scheme, where weights can be assigned to different categories based on how visually similar they are to the target object category. This is intuitive, as different categories will have varied degrees of similarity to a particular class, and some categories may have only a few (or many) visually similar classes. Thus, we modify Eq. (1) and define the transformation via visual similarity based on the proposed weighted nearest neighbor scheme as:
4.3 Knowledge Transfer via Semantic Relatedness
Following prior work [15, 41, 61], we observe that visual similarity is correlated with semantic relatedness. According to , this relationship is particularly strong when measurements are focused on the category instances themselves, ignoring image backgrounds. This observation is quite intriguing for object detection, where the main focus is on the target objects themselves. Hence, we draw on this fact and propose transferring knowledge from the natural language domain to help improve semi-supervised object detection.
Semantic similarity measure: As mentioned in Section 2.3, we use word embeddings to represent each category and to measure the semantic similarity between categories. Each of the categories is embedded as a word vector, more specifically a 300-dimensional word2vec embedding . Since each category is a WordNet  synset, we represent each category as the sum of the word vectors for each term in its synset, normalized to unit vector by its -norm. Out-of-vocabulary words are addressed by attempting to match case variants of the words (lowercase, Capitalized), _meaning:NTF . e.g _catcode:NTF a e.g. e.g., “aeroplane” is not in the vocabulary, but “Aeroplane” is. Failing that, we represent multiword phrases by the sum of the word vectors of each in-vocabulary word of the phrase, normalized to unit vector (“baby”+“bed” for baby bed). In several cases, we also augment synset terms with any category label defined in ILSVRC2013 that is not among the synset terms defined in WordNet (e.g. “bookshelf” for the WordNet synset bookcase, and “tv” and “monitor” for display).
Word embeddings often conflate multiple senses of a word into a single vector, leading to an issue with polysemous words. We observed this with many categories, for example seal (animal) is close to nail and tie (which, to further complicate matters, is actually meant to refer to its clothing sense); or the stationery ruler being related to lion. Since ILSVRC2013 categories are actually WordNet synsets, it makes perfect sense to exploit WordNet to help disambiguate the word senses. Thus, we integrate corpus-based representations with semantic knowledge from WordNet, by using AutoExtend  to encode the categories as synset embeddings in the original word2vec embedding space. AutoExtend exploits the interrelations between synsets, words and lexemes to learn an auto-encoder based on these constraints, as well as constraints on WordNet relations such as hypernyms (encouraging poodle and dog to have similar embeddings). We observed that AutoExtend has indeed helped form better semantic relations between the desired categories: seal is now clustered with other animal categories like whale and turtle, and the nearest neighbors for ruler are now rubber eraser, power drill and pencil box. In our detection experiments (Section 5), we found that while the ‘naive’ word embeddings performed better than the baselines, the synset embeddings yielded even better results. Thus, we concentrate on reporting the results of the latter.
We represent each category and with their synset embeddings, and compute the -norm of each pair as their semantic distance. The semantic similarity is inversely proportional to . We then transfer the semantic knowledge to the appearance model using Eq. (3) or its special case Eq. (1) as before.
As our semantic representations are in the form of vectors, we explore an alternative similarity measure as used in . We assume that each vector of a “weakly labeled” category (denoted as ) can be approximately represented by a linear combination of all the word vectors in : , where , and is a set of coefficients of the linear combination. We are motivated to find the solution which contains as few non-zero components as possible, since we tend to reconstruct category with fewer categories from (sparse representation). This optimal solution can be formulated as the following optimization:
Note that is a positive constraint on the coefficients, since negative components of sparse solutions for semantic transferring are meaningless: we only care about the most similar categories and not dissimilar categories. We solve Eq. (4) by using the positive constraint matching pursuit (PCMP) algorithm . Therefore, the final transformation via semantic transferring is formulated as:
where in the sparse representation case.
4.4 Mixture Transfer Model
We have proposed two different knowledge transfer models. Each of them can be integrated into the LSDA framework independently. In addition, since we consider the visual similarity at the whole image level and the semantic relatedness at object level, they can be combined simultaneously to provide complementary information. We use a simple but very effective combination of the two knowledge transfer models as our final mixture transfer model. Our mixture model is a linear combination of the visual similarity and the semantic similarity:
where is a function that takes the intersection of cooccurring categories between visual and sparse semantic related categories. is a parameter used to control the relative influence of the two similarity measurements. is set to 1 when only considering visual similarity transfer, and 0 for the semantic similarity transfer. We will analyze this parameter in Section 5.3.
4.5 Transfer on Bounding-box Regression
The detection windows generated by the region based detection models are the highest scoring proposals ( _meaning:NTF . e.g _catcode:NTF a e.g. e.g., Selective Search). In order to improve localization performance, a bounding-box regression stage  is commonly adopted to post-process the detection windows. This process needs bounding box annotations in training the regressors, which is an obstacle for “weakly labeled” categories in our case. Hence, we propose to transfer the class-specific regressors from “fully labeled” categories to “weakly labeled” categories based on the aforementioned similarity measures.
To train a regressor for each “fully labeled” category, we select a set of N training pairs , where is a vector indicating the center coordinates () of proposal together with ’s width and height (). is the corresponding ground-truth bounding box. Except where needed to avoid confusion we omit the super-script . The goal is to learn a mapping function which maps a region proposal to a ground-truth window . Each function within is modeled as a linear function of the features (namely the feature map after the last convolutional and pooling block of the ConvNet): , where is a vector of learnable parameters, is the feature of region proposal . can be learned by optimizing the following least squares objective function:
where is the regression target for the training pair () defined as:
The first two equations specify a scale-invariant translation of the center of the bounding box, while the remaining two specify the log-space translation of the width and height of the bounding box. After learning the parameters of the transformation function, a detection window (region proposal) can be transformed into a new prediction by applying:
The training pair () is selected if the proposal has maximum IoU overlap with ground-truth bounding box . The pair () is discarded if the maximum IoU overlap is less than a threshold (which is set to be 0.6 using a validation set).
For a “weakly labeled” category , the transformation function cannot be explicitly learned due to the absence of ground-truth bounding boxes. However, we can still transfer this knowledge from similar categories in the “fully labeled” subset :
where indicates any one of the aforementioned similarity measures.
|LSDA (only class invariant adaptation)||-||27.81||15.85||21.83|
||avg/weighted - 5||28.12 / –||15.97 / 16.12||22.05 / 22.12|
|LSDA (class invariant & specific adapt)||avg/weighted - 10||27.95 / –||16.15 / 16.28||22.05 / 22.12|
|avg/weighted - 100||27.91 / –||15.96 / 16.33||21.94 / 22.12|
|avg/weighted - 5||27.99 / –||17.42 / 17.59||22.71 / 22.79|
|Ours (visual transfer)||avg/weighted - 10||27.89 / –||17.62 / 18.41||22.76 / 23.15|
|avg/weighted - 100||28.30 / –||17.38 / 19.02||22.84 / 23.66|
|avg/weighted - 5||28.01 / –||17.32 / 17.53||22.67 / 22.77|
|Ours (semantic transfer)||avg/weighted - 10||28.00 / –||16.67 / 17.50||22.31 / 22.75|
|avg/weighted - 100||28.14 / –||17.04 / 18.32||23.23 / 23.28|
|Sparse rep. - 20||28.18||19.04||23.66|
Ours (mixture transfer )
|Ours (mixture transfer + BB reg.)||-||31.85||21.88||26.87|
|Oracle: Full Detection Network (no BB reg.)||-||29.72||26.25||28.00|
|Oracle: Full Detection Network (BB reg.)||-||32.17||29.46||30.82|
5.1 Dataset Overview
We investigate the proposed knowledge transfer models for large scale semi-supervised object detection on the ILSVRC2013 detection dataset covering 200 object categories. The training set is not exhaustively annotated because of its sheer size. There are also fewer annotated objects per training image than the validation and testing image (on average 1.53 objects for training vs. 2.5 objects for validation set). We follow all the experiment settings as in , and simulate having access to image-level annotations for all 200 categories and bounding box annotations only for the first 100 categories (alphabetical order). We separate the dataset into classification and detection sets. For the classification data, we use 200,000 images in total from all 200 categories of the training subset (around 1,000 images per category) and their image-level labels. The validation set is roughly split in half: val1 and val2 as in . For the detection training set, we take the images with their bounding boxes from only the first 100 categories () in val1 (around 5,000 images in total). Since the validation dataset is relatively small, we then augment val1 with 1,000 bounding box annotated images per class from the training set (following the same protocol of [4, 13]). Finally, we evaluate our knowledge transfer framework on the val2 dataset (9,917 images in total).
5.2 Implementation Details
, we first use the Caffe implementation of the “AlexNet” CNN. A pre-trained CNN on ILSVRC 2012 dataset is then fine-tuned on the classification training dataset (see Section 5.1). This CNN is then fine-tuned again for detection on the labeled region proposals of the first 100 categories (subset ) of val1. Selective Search  with “fast” mode is adopted to generate the region proposals from all the images in val1 and val2. We also report results using two deeper models of “VGG-Nets” , namely, the 16-layer model (VGG-16) and the 19-layer model (VGG-19), GoogLeNet and two ResNets (34-layer and 50-layer) with the Caffe toolbox. For the semantic representation, we use word2vec CBoW embeddings pre-trained on part of the Google News dataset comprising about 100 billion words . We train AutoExtend  using WordNet 3.0 to obtain synset embeddings, and using equal weights for the synset, lexeme and WordNet relation constraints (). As all categories are nouns, we use only hypernyms as the WordNet relation constraint. For the sparse representation of a target word vector in Eq. (4), we limit the maximum number of non-zero components to 20, since a target category has strong correlation with a small number of source categories. We set in Eq. (4) and in Eq. (7) based on a validation set. Other detailed information regarding training and detection can be found in Section 4.1.
5.3 Quantitative Evaluation on the “Weakly Labeled” Categories with “AlexNet”
Setting LSDA as the baseline, we compare the detection performance of our proposed knowledge transfer methods against LSDA. The results are summarized in Table I. As we are concerned with the detection of the “weakly labeled” categories, we focus mainly on the second column of the table (mean average precision (mAP) on ). Rows 1-5 in Table I are the baseline results for LSDA. The first row shows the detection results by applying a classification network (i.e., weakly supervised learning, and without adaptation) trained with only classification data, achieving only an mAP of 10.31% on the “weakly labeled” 100 categories. The last row shows the results of an oracle detection network which assumes that bounding boxes for all 200 categories are available (i.e., supervised learning). This is treated as the upper bound (26.25%) of the fully supervised framework. We observed that the best result obtained by LSDA is to adapt both category independent and category specific layers, and transforming with the weighted layer weight change of its 100 nearest neighbor categories (weighted-100 with 16.33% in Table I). Our “weighted” scheme works consistently better than its “average” counterpart.
For our visual knowledge transfer model, we show steady improvement over the baseline LSDA methods when considering the average weight change of both 5 and 10 visually similar categories, with 1.45% and 1.47% increase in mAP, respectively. This proves that our proposed visual similarity measure is superior to that of LSDA, showing that category-specific adaptation can indeed be improved based on knowledge about the visual similarities between categories. Further improvement is achieved by modeling individual weights of all 100 source categories according to their degree of visual similarities to the target category (weighted-100 with 19.02% in the table). This verifies our supposition that the transformation from a classifier to a detector of a certain category is more related to visually similar categories, and is proportional to their degrees of similarity. For example, motorcycle is most similar to bicycle. Thus the weight change from a bicycle classifier to detector has the largest influence on the transformation of motorcycle. The influence of less visually relevant categories, such as cart and chain saw, is much smaller. For visually dissimilar categories (apple, fig, hotdog, etc.), the influence is extremely insignificant. We show some examples of visual similarities between a target category and its source categories in the first row of Fig. 3. For each target category, the top-10 weighted nearest neighbor categories with their similarity degrees are visualized.
Our semantic knowledge transfer model also showed marked improvement over the LSDA baseline (Table I, Rows 9-12), and is comparable to the results of the visual transfer model. This suggests that the cross-domain knowledge transfer from semantic relatedness to visual similarity is very effective. The best performance for the semantic transfer model (19.04%) is obtained by sparsely reconstructing the target category with the source categories using the synset embeddings. We also compare the results of using other semantic similarity measures, as shown in Table II. The result of using synset embeddings (18.32%, using weighted-100, the same below) are superior to using ‘naive’ word2vec embeddings (17.83%) and WordNet based measures such as path-based similarity (17.08%) and Lin similarity  (17.31%). Several examples visualizing the related categories of the 10 largest semantic reconstruction coefficients are shown in the middle row of Fig. 3. We observe that semantic relatedness indeed correlates with visual similarity.
The state-of-the-art result using the 8-layer “Alex-Net” for semi-supervised detection on this dataset is achieved by our mixture transfer model which combines visual similarity and semantic relatedness. A boost in performance of 3.88% on original split (3.82%0.12%, based on 6 different splits of the dataset) is achieved over the best result reported by LSDA on the “weakly labeled” categories. We show examples of transferred categories with their corresponding weights for several target categories in the bottom row of Fig. 3. The parameter in Eq. (6) for the mixture model weights is set to 0.6 for final detection, where is chosen via cross-validation on the val1 detection set (Fig. 6). This suggests that the transferring of visual similarities is slightly more important than semantic relatedness, although both are indeed complementary. We do not tune for each category separately, though this can be expected to further improve our detection performance. Fig. 4 and Fig. 5 show some examples of correct and incorrect detections respectively. Although our proposed mixture transfer model achieves the state of the art in detecting the “weakly labeled” categories, it is still occasionally confused by visually similar categories.
5.4 Experimental Results with Deeper Neural Networks
Previous work [64, 67, 66] found that region based CNN detection performance is significantly influenced by the choice of CNN architecture. In Table III, we show some detection results using the 16-layer and 19-layer deep “VGG-Nets” proposed by Simonyan and Zisserman , “GoogLeNet” (Inception-v2), together with the 34-layer and 50-layer “ResNets” . The VGG-16 network consists of 13 convolutional layers of very small (
) convolution filters, with 5 max pooling layers interspersed, and topped with 3 fully connected layers (namely,, and ). The VGG-19 network extends VGG-16 by inserting 3 more convolutional layers, while keeping other layer configurations unchanged. The state-of-the-art residual networks (ResNets) make use of identity shortcut connections that enable flow of information across layers without decay. In the aforementioned deep neural nets, we transfer the parameters of the last fully connected layer ( layer for VGG-Nets and the only layer in GoogLeNet and ResNets).
As can be seen from Table III, the very deep ConvNets VGG-16, VGG-19 and GoogLeNet significantly outperform Alex-Net for all the adaptation methods. Our knowledge transfer models using the very deep VGG-Nets with different similarity measures show consistent improvement over the LSDA baseline method. The relative overall improvement over performance using VGG-Nets is similar with that of AlexNet. GoogLeNet obtains similar results to VGG-19, while the best performance is achieved by ResNet-50.
5.5 Experimental Results with Bounding-box Regression
The results in Table I and Table III show that the transferred bounding-box regression from “fully labeled” categories fixes a large number of incorrectly localized detections, boosting mAP by about 2 points for the “weakly labeled” categories. The bounding-box regression process could boost mAP by 3 to 4 points if the bounding box annotations for all the categories were provided. We show some example detections before and after bounding box regression on the “weakly labeled” categories in Fig. 7, using VGG-16.
In addition to the results reported above using the default Intersection-over-Union (IoU) threshold , we evaluate detection performance by setting different IoU overlap ratios before and after bounding box regression using the best performing network, _meaning:NTF . i.e _catcode:NTF a i.e. i.e., ResNet-50. The mAP@IoU=0.6 is 23.21 v.s. 26.02 before and after regression, mAP@IoU=0.7 is 17.89 v.s. 20.64, respectively. These results validate that the proposed regression transfer is very effective in moving bounding box boundaries so as to cover more foreground object regions, by transferring the class-specific regressors from “fully labeled” categories to “weakly labeled” categories based on the proposed similarity measures. Note that results in other parts of this paper are reported as mAP@IoU=0.5, unless specified otherwise.
5.6 Experimental Results with Fast R-CNN
The proposed knowledge transfer method is applied to Fast R-CNN, without modifying much of the framework. Our Fast R-CNN based transfer framework is much faster than the R-CNN based approach, since in Fast R-CNN, in Fast R-CNN, an image is first fed into a CNN to create a convolution feature map and a single feature vector is then extracted from a Region of Interest (RoI) pooling layer for each region proposal, while in R-CNN, each region proposal in an image is fed into a CNN to extract feature independently, which is considerably more computationally expensive than Fast R-CNN.
We investigate two different bounding box regression strategies in Fast R-CNN. For the first strategy, we remove the built-in bounding-box regression layer in the Fast R-CNN pipeline and transfer the regressor off-line after detection as in Section 4.5, like the SPP-Net, which we call “2-stage” Fast R-CNN. For the second strategy, we use the built-in bounding-box regression layer, which is actually a class-specific fully-connected layer with neurons (4 indicates the coordinates of a bounding box position, indicates the number of categories). This class-specific layer can be therefore transferred from “fully labeled” categories to “weakly labeled” categories using the proposed similarity measures in a way similar to that described in Section 4.5. This strategy is called “end-to-end”. Table IV shows the detection performance using these Fast R-CNN based ConvNets. As can be seen from the table, Fast R-CNN achieves consistently better performance over the R-CNN based approach, and the “end-to-end” joint training/testing is superior to the “2-stage” pipeline. State-of-the-art detection performance on the weakly labeled categories (29.71) is obtained by Fast R-CNN based ResNet-50, which is very close to that of the fully supervised R-CNN based AlexNet (30.82).
In this paper, we investigated how knowledge about object similarities from both visual and semantic domains can be transferred to adapt an image classifier to an object detector in a semi-supervised setting. We experimented with different CNN architectures, found clear evidence that both visual and semantic similarities play an essential role in improving the adaptation process, and that the combination of the two modalities yielded state-of-the-art performance, suggesting that knowledge inherent in visual and semantic domains is complementary. Future work includes extracting more knowledge from different domains, using better representations, and investigating the possibility of using category-invariant properties, _meaning:NTF . e.g _catcode:NTF a e.g. e.g., the difference between feature distributions of whole images and target objects, to help knowledge transfer. We believe that the combination of knowledge from different domains is key to improving semi-supervised object detection.
This work was partly supported by the French Research Agency, Agence Nationale de Recherche (ANR), through the VideoSense Project under Grant 2009 CORD 026 02, and the Visen project under Grants ANR-12-CHRI-0002-04 and UK EPSRC EP/K019082/1 within the framework of the ERA-Net CHIST-ERA, and by the Partner University Foundation through the 4D Vision project. The authors thank NVIDIA for providing the Titan X GPU.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Neural Information Processing Systems (NIPS), 2012.
-  C. Szegedy, A. Toshev, and D. Erhan, “Deep neural networks for object detection,” in Neural Information Processing Systems (NIPS), 2013, pp. 2553–2561.
-  P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “Overfeat: Integrated recognition, localization and detection using convolutional networks,” in International Conference on Learning Representations (ICLR), 2014.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies
for accurate object detection and semantic segmentation,” in
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 37, no. 9, pp. 1904–1916, 2015.
-  R. Girshick, “Fast R-CNN:,” in International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.
-  S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Neural Information Processing Systems (NIPS), 2015, pp. 91–99.
-  J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. Berg, “SSD: Single shot multibox detector,” in European Conference on Computer Vision (ECCV), 2016, pp. 21–37.
-  M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman, “The Pascal visual object classes (VOC) challenge,” International Journal of Computer Vision (IJCV), vol. 88, no. 2, pp. 303–338, 2010.
-  O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV), vol. 0, no. 0, pp. 1–42, April 2015.
-  T. Lin, M. Maire, S. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in European Conference on Computer Vision (ECCV), 2014, pp. 740–755.
-  J. Hoffman, S. Guadarrama, E. Tzeng, R. Hu, J. Donahue, R. Girshick, T. Darrell, and K. Saenko, “LSDA: Large scale detection through adaptation,” in Neural Information Processing Systems (NIPS), 2014.
-  B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object detectors emerge in deep scene CNNs,” in International Conference on Learning Representations (ICLR), 2015.
-  T. Deselaers and V. Ferrari, “Visual and semantic similarity in imagenet,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
-  Y. Tang, J. Wang, B. Gao, E. Dellandrea, R. Gaizauskas, and L. Chen, “Large scale semi-supervised object detection using visual and semantic knowledge transfer,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  D. Crandall and D. Huttenlocher, “Weakly supervised learning of part-based spatial models for visual object recognition,” in European Conference on Computer Vision (ECCV), 2006.
-  O. Chum and A. Zisserman, “An exemplar model for learning object classes,” in 2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp. 1–8.
-  C. Galleguillos, B. Babenko, A. Rabinovich, and S. Belongie, “Weakly supervised object recognition and localization with stable segmentations,” in European Conference on Computer Vision (ECCV), 2008.
-  M. Nguyen, L. Torresani, F. de la Torre, and C. Rother, “Weakly supervised discriminative localization and classification: a joint learning process,” in International Conference on Computer Vision (ICCV), 2009, pp. 1925–1932.
-  P. Siva and T. Xiang, “Weakly supervised object detector learning with model drift detection,” in International Conference on Computer Vision (ICCV), 2011.
M. Pandey and S. Lazebnik, “Scene recognition and weakly supervised object localization with deformable part-based models,” inInternational Conference on Computer Vision (ICCV), 2011.
-  P. Siva, C. Russell, and T. Xiang, “In defence of negative mining for annotating weakly labelled data,” in European Conference on Computer Vision (ECCV), 2012.
-  T. Deselaers, B. Alexe, and V. Ferrari, “Weakly supervised localization and learning with generic knowledge,” International Journal of Computer Vision (IJCV), vol. 100, no. 3, pp. 275–293, 2012.
-  Z. Shi, T. M. Hospedales, and T. Xiang, “Bayesian joint topic modelling for weakly supervised object localisation,” in International Conference on Computer Vision (ICCV), 2013, pp. 2984–2991.
-  Y. Tang, X. Wang, E. Dellandrea, S. Masnou, and L. Chen, “Fusing generic objectness and deformable part-based models for weakly supervised object detection,” in IEEE International Conference on Image Processing (ICIP), 2014.
-  H. Bilen, M. Pedersoli, and T. Tuytelaars, “Weakly supervised object detection with posterior regularization,” in British Machine Vision Conference (BMVC), 2014.
H. O. Song, R. Girshick, S. Jegelka, J. Mairal, Z. Harchaoui, and T. Darrell,
“On learning to localize objects with minimal supervision,” in
International Conference on Machine Learning (ICML), 2014.
-  H. Bilen, M. Pedersoli, and T. Tuytelaars, “Weakly supervised object detection with convex clustering,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
-  C. Wang, K. Huang, W. Ren, J. Zhang, and S. Maybank, “Large-scale weakly supervised object localization via latent category learning,” IEEE Transactions on Image Processing (TIP), vol. 24, no. 4, pp. 1371–1385, April 2015.
-  Y. Tang, X. Wang, E. Dellandrea, and L. Chen, “Weakly supervised learning of deformable part-based models for object detection via region proposals,” IEEE Transactions on Multimedia (TMM), vol. PP, no. 99, pp. 1–1, 2016.
-  J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, “Selective search for object recognition,” International Journal of Computer Vision (IJCV), vol. 104, no. 2, pp. 154–171, 2013.
-  C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in European Conference on Computer Vision (ECCV), 2014.
-  M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Is object localization for free? - weakly-supervised learning with convolutional neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
-  H. Bilen and A. Vedaldi, “Weakly supervised deep detection networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
B. Zhou, A. Khosla, L. A., A. Oliva, and A. Torralba, “Learning Deep Features for Discriminative Localization.”IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  L. Shao, F. Zhu, and X. Li, “Transfer learning for visual categorization: A survey,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS), vol. 26, no. 5, pp. 1019–1034, May 2015.
-  J. Donahue, J. Hoffman, E. Rodner, K. Saenko, and T. Darrell, “Semi-supervised domain adaptation with instance constraints,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
-  M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
-  M. Rochan and Y. Wang, “Weakly supervised localization of novel objects using appearance transfer,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
-  X. Shu, G.-J. Qi, J. Tang, and J. Wang, “Weakly-shared deep transfer networks for heterogeneous-domain knowledge propagation,” in ACM International Conference on Multimedia (MM), 2015.
Y. Zhu, Y. Chen, Z. Lu, S. J. Pan, G.-R. Xue, Y. Yu, and Q. Yang,
“Heterogeneous transfer learning for image classification.” in
AAAI Conference on Artificial Intelligence (AAAI), 2011.
-  Y. Lu, L. Chen, A. Saidi, E. Dellandrea, and Y. Wang, “Discriminative transfer learning using similarities and dissimilarities,” IEEE Transactions on Neural Networks and Learning Systems, vol. PP, no. 99, pp. 1–14, 2017.
-  K. K. Singh, F. Xiao, and Y. J. Lee, “Track and transfer: Watching videos to simulate strong human supervision for weakly-supervised object detection,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  A. Frome, G. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” in Neural Information Processing Systems (NIPS), 2013.
-  Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Neural Information Processing Systems (NIPS), 2007.
-  C. Fellbaum, Ed., WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press, 1998.
-  C. Leacock and M. Chodorow, “Combining local context and WordNet similarity for word sense identification,” in WordNet: An Electronic Lexical Database, C. Fellbaum, Ed. MIT Press, 1998, pp. 265–283.
-  P. Resnik, “Using information content to evaluate semantic similarity in a taxonomy,” in Proceedings of the International Joint Conference for Artificial Intelligence (IJCAI-95), 1995.
-  D. Lin, “An information-theoretic definition of similarity,” in International Conference on Machine Learning (ICML), 1998.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” inNeural Information Processing Systems (NIPS), 2013.
-  J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543.
-  T. Mikolov, W.-t. Yih, and G. Zweig, “Linguistic regularities in continuous space word representations,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2013.
-  I. Misra, A. Shrivastava, and M. Hebert, “Watch and learn: Semi-supervised learning of object detectors from videos,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
-  C. Rosenberg, M. Hebert, and H. Schneiderman, “Semi-supervised self-training of object detection models,” in IEEE Workshops on Application of Computer Vision (WACV), 2005.
-  Y. Yang, G. Shu, and M. Shah, “Semi-supervised learning of feature hierarchies for object detection in a video,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
-  P. Agrawal, R. Girshick, and J. Malik, “Analyzing the performance of multilayer neural networks for object recognition,” in European Conference on Computer Vision (ECCV), 2014.
-  Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in ACM International Conference on Multimedia (MM), 2014.
-  M. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European Conference on Computer Vision (ECCV), 2014.
-  M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych, and B. Schiele, “What helps where – and why? semantic relatedness for knowledge transfer,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
-  S. Rothe and H. Schütze, “Autoextend: Extending word embeddings to embeddings for synsets and lexemes,” in the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL), 2015, pp. 1793–1803.
-  B. Gao, E. Dellandrea, and L. Chen, “Music sparse decomposition onto a midi dictionary of musical words and its application to music mood classification,” in International Workshop on Content-Based Multimedia Indexing (CBMI), 2012.
-  K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations (ICLR), 2015.
-  C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
-  R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Region-based convolutional networks for accurate object detection and segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 38, no. 1, pp. 142–158, 2016.