DeepAI
Log In Sign Up

Object localization in ImageNet by looking out of the window

01/06/2015
by   Alexander Vezhnevets, et al.
0

We propose a method for annotating the location of objects in ImageNet. Traditionally, this is cast as an image window classification problem, where each window is considered independently and scored based on its appearance alone. Instead, we propose a method which scores each candidate window in the context of all other windows in the image, taking into account their similarity in appearance space as well as their spatial relations in the image plane. We devise a fast and exact procedure to optimize our scoring function over all candidate windows in an image, and we learn its parameters using structured output regression. We demonstrate on 92000 images from ImageNet that this significantly improves localization over recent techniques that score windows in isolation.

READ FULL TEXT VIEW PDF

page 2

page 5

page 10

03/06/2022

Smoothing with the Best Rectangle Window is Optimal for All Tapered Rectangle Windows

We investigate the optimal selection of weight windows for the problem o...
12/11/2014

An active search strategy for efficient object class detection

Object class detectors typically apply a window classifier to all the wi...
12/11/2013

Associative embeddings for large-scale knowledge transfer with self-assessment

We propose a method for knowledge transfer between semantically related ...
11/02/2021

Design of Tight Minimum-Sidelobe Windows by Riemannian Newton's Method

The short-time Fourier transform (STFT), or the discrete Gabor transform...
10/23/2014

Density-Based Region Search with Arbitrary Shape for Object Localization

Region search is widely used for object localization. Typically, the reg...
11/16/2022

An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022

This technical report describes the CONE approach for Ego4D Natural Lang...
10/17/2019

Introducing Hann windows for reducing edge-effects in patch-based image segmentation

There is a limitation in the size of an image that can be processed usin...

1 Introduction

The ImageNet database [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-fei] contains over 14 million images annotated by the class label of the main object they contain. However, only a fraction of them have bounding-box annotations (). Automatically annotating object locations in ImageNet is a challenging problem, which has recently drawn attention [Guillaumin et al.(2014)Guillaumin, Küttel, and Ferrari, Guillaumin and Ferrari(2012), Vezhnevets and Ferrari(2014)]. These annotations could be used as training data for problems such as object class detection [Dalal and Triggs(2005)], tracking [Leibe et al.(2007)Leibe, Schindler, and Van Gool]

and pose estimation 

[Andriluka et al.(2010)Andriluka, Roth, and Schiele]. Traditionally, object localization is cast as an image window scoring problem, where a scoring function is trained on images with bounding-boxes and applied to ones without. The image is first decomposed into candidate windows, typically by object proposal generation [Alexe et al.(2012)Alexe, Deselaers, and Ferrari, Manen et al.(2013)Manen, Guillaumin, and Van Gool, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Dollar and Zitnick(2014)]

. Each window is then scored by a classifier trained to discriminate instances of the class from other windows 

[Guillaumin and Ferrari(2012), Wang et al.(2013)Wang, Yang, Zhu, and Lin, Dalal and Triggs(2005), Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik, Harzallah et al.(2009)Harzallah, Jurie, and Schmid, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Vedaldi et al.(2009)Vedaldi, Gulshan, Varma, and Zisserman] or a regressor trained to predict their overlap with the object [Vezhnevets and Ferrari(2014), Blaschko and Lampert(2008), Vedaldi and Zisserman(2009), Lehmann et al.(2011)Lehmann, Gehler, and Van Gool]. Highly scored windows are finally deemed to contain the object. In this paradigm, the classifier looks at one window at a time, making a decision based only on that window’s appearance.

Figure 1: Connecting the appearance and window position spaces. (a) a window tight on the baseball (green star in the appearance space plot) and some larger windows containing it (red circles in the appearance space). Black points in appearance space represent all other candidate windows. (b) all windows with high overlap with the wolf are shown in red, both in the image and in appearance space. The ground-truth bounding-box to be found is shown in green. The appearance space plots are actual datapoints, representing windows in 3-dimensional Associative Embedding of SURF bag-of-words. Please see main text for discussion.

We believe there is more information in the collection of windows in an image. By taking into account the appearance of all windows at the same time and connecting it to their spatial relations in the image plane, we could go beyond what can be done by looking at one window at a time. Consider the baseball in fig. 1. For a traditional method to succeed, the appearance classifier needs to score the window on the baseball higher than the windows containing it. The container windows cannot help except by scoring lower and be discarded. By considering one window at a time with a classifier that only tries to predict whether it covers the object tightly, one cannot do much more than that. The first key element of our work is to predict richer spatial relations between each candidate window and the object to be detected, including part and container relations. The second key element is to employ these predictions to reason about relations between different windows. In this example, the container windows are predicted to contain a smaller target object somewhere inside them, and thereby actively help by reinforcing the score of the baseball window. Fig. 1 illustrates another example of the benefits of analyzing all windows jointly. Several windows which have high overlap with each other and with the wolf form a dense cluster in appearance space, making it hard to discriminate the precise bounding-box by its appearance alone. However, the precise bounding-box is positioned at an extreme point of the cluster — on the tip. By considering the configuration of all the windows in appearance space together we can reinforce its score.

In a nutshell, we propose to localize objects in ImageNet by scoring each candidate window in the context of all other windows in the image, taking into account their similarity in appearance space as well as their spatial relations in the image plane. To represent spatial relations of windows we propose a descriptor indicative of the part/container relationship of the two windows and of how well aligned they are (sec. 2). We learn a windows appearance similarity kernel using the recent Associative Embedding technique [Vezhnevets and Ferrari(2014)] (sec. 3). We describe each window with a set of hyper-features connecting the appearance similarity and spatial relations of that window to all other windows in the same image. These hyper-features are indicative of the object’s presence when the appearance of a window alone is not enough (e.g. fig 1). These hyper-features are then linearly combined into an overall scoring function (sec. 4). We devise a fast and exact procedure to optimize our scoring function over all candidate windows in a test image (sec. 4.1), and we learn its parameters using structured output regression [Tsochantaridis et al.(2005)Tsochantaridis, Joachims, Hofmann, and Altun] (sec. 4.2).

We evaluate our method on a subset of ImageNet containing 219 classes with more than 92000 images [Guillaumin and Ferrari(2012), Vezhnevets and Ferrari(2014), Guillaumin et al.(2014)Guillaumin, Küttel, and Ferrari]. The experiments show that our method outperforms a recent approach for this task [Vezhnevets and Ferrari(2014)], an MKL-SVM baseline [Vedaldi et al.(2009)Vedaldi, Gulshan, Varma, and Zisserman] based on the same features, and the popular UVA object detector [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders]. The remainder of the paper is organized as follows. Sec. 2 and 3 introduce the spatial relation descriptors which we use in sec. 4 to define our new localization model. In sec. 5 we review related work. Experiments and conclusions are presented in sec. 6.

2 Describing spatial relations between windows

Candidate windows.

Recently, object class detectors are moving away from the sliding-window paradigm and operate instead on a relatively small collection of candidate windows [Alexe et al.(2012)Alexe, Deselaers, and Ferrari, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik, Guillaumin and Ferrari(2012), Vezhnevets and Ferrari(2014), Manen et al.(2013)Manen, Guillaumin, and Van Gool, Dollar and Zitnick(2014), Wang et al.(2013)Wang, Yang, Zhu, and Lin] (also called ‘object proposals’). The candidate window generators are designed to respond to objects of any class, and typically just candidates are sufficient to cover all objects in a cluttered image [Alexe et al.(2012)Alexe, Deselaers, and Ferrari, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Manen et al.(2013)Manen, Guillaumin, and Van Gool, Dollar and Zitnick(2014)]. Given a test image, the object localization task is then to select a candidate window covering an instance of a particular class (e.g. cars). Following this trend, we generate about 1000 candidate windows using the recent method [Manen et al.(2013)Manen, Guillaumin, and Van Gool].

Figure 2: Spatial relations between two windows. (a) The first element indicate how much and overlap, in the traditional PASCAL VOC sense [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman]. The second element indicates whether window is a part of . The third element measures whether is a container of . (b) Computing the spatial relations and for hyper-features (3) and (4) .

Spatial relation descriptor.

We introduce here a representation of the spatial relations between two windows and , which we later use in our localization model (sec. 4). We summarize the spatial relation between windows and using the following spatial relation descriptor (fig. 2)

(1)

where the operator indicates the area of the intersection between the windows, and the area of their union. The descriptor captures three different kinds of spatial relations. The first is the familiar intersection-over-union (overlap), which is often used to quantify the accuracy of an object detector [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman, ils(2011)]. It is when , and decays rapidly with the misalignment between the two windows. The second relation measures how much of is contained inside . It is high when is a part of , e.g. when is a car and is a wheel. The third relation measures how much of is contained inside . It is high when contains , e.g. is a snooker ball and is a snooker table. All three relations are if and are disjoint and are if and match perfectly. Hence the descriptor is indicative for part/container relationships of the two windows and of how well aligned they are.

Vector field of window relations.

Relative to a particular candidate window , we can compute the spatial relation descriptor to any window

. This induces a vector field

over the continuous space of all possible window positions. We observe the field only at the discrete set of candidate windows . A key element of our work is to connect this field of spatial relations to measurements of appearance similarity between windows. This connection between position and appearance spaces drives the new components in our localization model (sec. 4).

3 Predicting spatial relations with the object

A particularly interesting case is when is the true bounding-box of an object . For the images in the training set, we know the spatial relations between all candidate windows and the bounding-box . We can use them to learn to predict the spatial relation between candidate windows and the object from window appearance features in a test image, where ground-truth bounding-boxes are not given.

Following [Vezhnevets and Ferrari(2014)], we use Gaussian Processes regression (GP) [Rasmussen and Williams(2005)]

to learn to predict a probability distribution

for each spatial relation given window appearance features . We use zero mean and learn the kernel (covariance function) as in [Vezhnevets and Ferrari(2014)]. This kernel plays the role of an appearance similarity measure between two windows. The GP learns kernel parameters so that the resulting appearance similarity correlates with the spatial relation to be predicted, i.e. so that two windows which have high kernel value also have a similar spatial relation to the ground-truth. We will use the learnt later in sec. 4.

For a window

in a test image, the GP predicts a Gaussian distribution for each relation descriptors. We denote the means of these predictive distributions as

, and their standard deviation as

. The standard deviation is the same for all relations, as we use the same kernel and inducing points.

4 Object localization with spatial relations

We are given a test image with (a) set of candidate windows ; (b) their appearance features ; (c) the mean and standard deviation of their spatial relations with the object bounding-box, as predicted by the GP; (d) the appearance similarity kernel (sec. 3).

Let be a candidate window to be scored. We proceed by defining a set of hyper-features characterizing , and then define our scoring function through them.

Consistency of predicted & induced spatial relations

(2)

Assume for a moment that

correctly localizes an instance of the object class. Selecting would induce spatial relations to all other windows . The hyper-feature checks whether these induced spatial relations are consistent with those predicted by GP based on the appearance of the other windows (). If so, that is a good sign that is indeed the correct location of the object. More precisely, the hyper-feature measures the disagreement between the induced and predicted on the window with the largest disagreement. Fig. 3 illustrates it on a toy example. The maximum disagreement, instead of a more intuitive mean, is less influenced by disagreement over background windows, which are usually predicted by GP to have small, but non-zero relations to the object. It focuses better on the alignment of the peaks of the predicted and observed measurements of the vector field , which is more indicative of being a correct localization.

Global spatial relations & appearance

(3)

This hyper-feature reacts to pairs of candidate windows with similar appearance (high ) but different spatial relations to . Two windows contribute significantly to the sum if they look similar (high ) and their spatial relations to are different (high .

A large value of indicates that the vector field of the spatial relations to is not smooth with respect to appearance similarity. This indicates that has a special role in the structure of spatial and appearance relations within that image. By measuring this pattern, helps the localization algorithm to select a better window, when the information contained in appearance features of alone is not enough. For example, a window tightly covering a small object such as the baseball in fig. 1 has high , because other windows containing it often look similar to windows not containing it. In this case, a high value of is a positive indication for being a correct localization. On the other hand, a window tightly covering the wolf in fig. 1 has low , because windows that overlap with it are all similar to each other in appearance space. In this case, this low value is a positive indication for being correct. In which direction to use this hyper-feature is left to the learning of its weight in the full scoring function, which is separate for each object class (sec 4.2).

Figure 3: Hyper-features for a window are computed by finding a maximum disagreement between spatial relations that are predicted by GP and induced by spatial relations for all other windows in the image. The figure illustrates this on a toy example with three windows . For each the pairs of and with maximum disagreement are highlight in red.

Local spatial relations & appearance

(4)

This hyper-feature is analogue to , but focuses around in appearance space. It is indicative of whether windows that look similar to (high ) are also similar in position in the image, i.e. their spatial relation to is close to .

Window classifier score .

The last hyper-feature is the score of a classifier which predicts whether a window covers an instance of the class, based only on its appearance features . Standard approaches to object localization typically consider only this cue [Guillaumin and Ferrari(2012), Wang et al.(2013)Wang, Yang, Zhu, and Lin, Dalal and Triggs(2005), Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik, Harzallah et al.(2009)Harzallah, Jurie, and Schmid, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Vedaldi et al.(2009)Vedaldi, Gulshan, Varma, and Zisserman]. In principle, we could use any such method as the score here. In practice, we reuse the GP prediction of the overlap of a window with the object bounding-box as . One possibility would be to simply use the mean predicted overlap . However, as shown in  [Vezhnevets and Ferrari(2014)], it is beneficial to take into account the uncertainty of the predicted overlap, which is also provided by the GP as the standard deviation of the estimate

(5)

Using this hyper-features alone would correspond to the method of [Vezhnevets and Ferrari(2014)].

Complete score.

Let

be a concatenation of all hyper-features defined above for a particular candidate window , over all three possible relations: . This amounts to 11 features in total. We formulate the following score function

(6)

where is the scalar product. Object localization translates to solving
over all candidate windows in the test image. The vector of scalars parametrizes the score function by weighting the hyper-features (possibly with a negative weight). We show how to efficiently maximize over in sec. 4.1 and how to learn in sec. 4.2.

4.1 Fast inference

We can maximize the complete score (6) over simply by evaluating it for all possible and picking the best one. The most computationally expensive part is the hyper-feature (3). For a given , it sums over all pairs of candidate windows, which requires subtractions and multiplications. Thus, a naive maximization over all costs .

To simplify the notation, here we write the score function with only one argument , as the other arguments are fixed during maximization. Note that , therefore

(7)

Where is the weight of hyper-feature . By substituting the elements in the sum over pairs in (3) with the bound (7), we obtain an upper bound on the term of the score (in fact three bounds, one for each ). We can then obtain an upper bound on the full score by computing all other hyper-features and adding them to the bounds on . This upper bound is fast to compute, as (7) only depends on appearance features , not on , and computing the other hyper-features is linear in .

We use the bound in an early rejection algorithm. We form a queue by sorting windows in descending order of . We then evaluate the full score of the first in the queue and store it as the current maximum. We then go to the next in the queue. If its upper bound is smaller than the current maximum, then we discard it without computing its full score. Otherwise, we compute and set it as the current maximum if it is better than the previous best one. We iteratively go through the queue and at the end return the current maximum (which is now the best over all ). Notice, that the proposed fast inference method is exact: it outputs the same solution as brute force evaluation of all possible .

4.2 Learning with structured output regression

The scoring function (6) is parametrized by the weight vector . We learn an from the training data of each class using a structured output regression formulation [Blaschko and Lampert(2008), Lampert et al.(2008)Lampert, Blaschko, and Hofmann, Vedaldi and Zisserman(2009), Lehmann et al.(2011)Lehmann, Gehler, and Van Gool]. Ideally, we look for so that, for each training image , the candidate window that best overlaps with the ground-truth bounding-box has the highest score. It is also good to encourage the score difference between the best window and any other window to be proportional to their overlap. This makes the learning problem smoother and better behaved than when using a naive loss which equally penalizes all windows other than . Hence, we use the loss proposed by [Blaschko and Lampert(2008)], which formalizes this intuition. We can find by solving the following optimization problem

(8)

where indexes over all training images. This is a convex optimization problem, but it has hundred of thousands of constraints (i.e. about candidate windows for each training image, times about training images per class in our experiments). We solve it efficiently using quadratic programming with constraint generation [Bakir et al.(2007)Bakir, Schölkopf, Smola, Taskar, Vishwanathan, and S.V.N]. This involves finding the most violated constraint for a current . We do this exactly as we can solve the inference problem (6) and the loss decomposes into a sum of terms which depend on a single window. Thanks to this, the constraint generation procedure will find the global optimum of (8[Bakir et al.(2007)Bakir, Schölkopf, Smola, Taskar, Vishwanathan, and S.V.N].

5 Related work

The first work to try to annotate object locations in ImageNet [Guillaumin and Ferrari(2012)] addressed it as a window classification problem, where a classifier is trained on annotated images is then used to classify windows in images with no annotation. Later [Vezhnevets and Ferrari(2014)] proposed to regress the overlap of a window with the object using GP [Rasmussen and Williams(2005)], which allowed for self-assessment thanks to GP probabilistic output. We build on [Vezhnevets and Ferrari(2014)], using Associative Embedding to learn the kernel between windows appearance features and GP to predict the spatial relations between a window and the object. Note how the model of [Vezhnevets and Ferrari(2014)] is equivalent to ours when using only the hyper-feature.

Context.

The seminal work of Torralba [Torralba(2003)] has shown that global image descriptors such as GIST give a valuable cue about which classes might be present in an image (e.g. indoor scenes are likely to have TVs, but unlikely to have cars). Since then, many object detectors [Murphy et al.(2003)Murphy, Torralba, and Freeman, Russell et al.(2007)Russell, Torralba, Liu, Ferugs, and Freeman, Harzallah et al.(2009)Harzallah, Jurie, and Schmid, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, ils(2013), Song et al.(2011)Song, Chen, Huang, Hua, and Yan] have employed such global context to re-score their detections, thereby removing out-of-context false-positives. Some of these works also incorporate the region surrounding the object into the window classifier to leverage local context [Dalal and Triggs(2005), Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Li et al.(2011)Li, Parikh, and Chen, Mottaghi et al.(2014)Mottaghi, Chen, Liu, Cho, Lee, Fidler, Urtasun, and Yuille].

Other works [Rabinovich et al.(2007)Rabinovich, Vedaldi, Galleguillos, Wiewiora, and Belongie, Heitz and Koller(2008), Choi et al.(2010)Choi, Lim, Torralba, and Willsky, Desai et al.(2009)Desai, Ramanan, and Folkess] model context as the interactions between multiple object classes in the same image. Rabinovich et al. [Rabinovich et al.(2007)Rabinovich, Vedaldi, Galleguillos, Wiewiora, and Belongie] use local detectors to first assign a class label to each segment in the image and then adjusts these labels by taking into account co-occurrence between classes. Heitz and Koller [Heitz and Koller(2008)] exploits context provided by "stuff" (background classes like road) to guide the localization of "things" (objects like cars). Several works [Choi et al.(2010)Choi, Lim, Torralba, and Willsky, Desai et al.(2009)Desai, Ramanan, and Folkess] model the co-occurrence and spatial relations between object classes in the training data and use them to post-process the output of individual object class detectors. An extensive empirical study of different context-based methods can found in [Divvala et al.(2009)Divvala, Hoiem, Hays, Efros, and Hebert].

The force driving those works is the semantic and spatial structure of scenes as arrangements of different object classes in particular positions. Instead, our technique works on a different level, improving object localization for a single class by integrating cues from the appearance and spatial relations of all windows in an image. It can be seen as a new, complementary form of context.

Localization with structured output regression

was first proposed by [Blaschko and Lampert(2008)]. They devised a training strategy that specifically optimizes localization accuracy, by taking into account the overlap of training image windows with the ground-truth. They try to learn a function which scores windows with high overlap with ground-truth higher than those with low overlap. The approach was extended by [Vedaldi and Zisserman(2009)] to include latent variables for handling multiple aspects of appearance and truncated object instances. At test time an efficient branch-and-bound algorithm is used to find the window with the maximum score. Branch-and-bound methods for localization where further explored in [Lampert et al.(2008)Lampert, Blaschko, and Hofmann, Lehmann et al.(2011)Lehmann, Gehler, and Van Gool].

Importantly, the scoring function in [Blaschko and Lampert(2008), Lampert et al.(2008)Lampert, Blaschko, and Hofmann, Vedaldi and Zisserman(2009), Lehmann et al.(2011)Lehmann, Gehler, and Van Gool] still scores each window in the test image independently. In our work instead we score each window in the context of all other windows in the image, taking into account their similarity in appearance space as well as their spatial relations in the image plane (sec. 4). We also use structured output regression [Tsochantaridis et al.(2005)Tsochantaridis, Joachims, Hofmann, and Altun] for learning the parameters of our scoring function (sec. 4.2). However, due to interaction between all windows in the test image, our maximization problem is more complex than in [Blaschko and Lampert(2008), Lampert et al.(2008)Lampert, Blaschko, and Hofmann], making their branch-and-bound method inapplicable. Instead, we devise an early-rejection method that uses the particular structure of our scoring function to reduce the number of evaluations of its most expensive terms (sec. 4.1).

6 Experiments and conclusions

We perform experiments on the subset of ImageNet [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-fei] defined by [Guillaumin and Ferrari(2012), Vezhnevets and Ferrari(2014), Guillaumin et al.(2014)Guillaumin, Küttel, and Ferrari], which consists of 219 classes for a total of 92K images with ground-truth bounding-boxes. Following [Vezhnevets and Ferrari(2014)], we split them in two disjoint subsets of 60K and 32K for training and testing respectively. The classes are very diverse and include animals as well as man-made objects (fig. 5). The task is to localize the object of interest in images known to contain a given class [Guillaumin and Ferrari(2012), Vezhnevets and Ferrari(2014), Guillaumin et al.(2014)Guillaumin, Küttel, and Ferrari]. We train a separate model for each class using the corresponding images from the training set.

Features.

For our method and all the baselines we use the same features as AE-GP+ method from [Vezhnevets and Ferrari(2014)]: (i) three ultra-dense SIFT bag-of-words histograms on different color spaces [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders] (each 36000 dimensions); (ii) a SURF bag-of-word from [Vezhnevets and Ferrari(2014)] (17000 dimensions); (iii) HOG [Dalal and Triggs(2005)] (2048 dimensions). We embed each feature type separately in a 10-dimensional AE [Vezhnevets and Ferrari(2014)] space. Next, we concatenate them together and add location and scale features as in [Guillaumin and Ferrari(2012), Vezhnevets and Ferrari(2014)]. In total, this leads to a 54-dimensional space on which the GP operates, i.e. only 54 parameters to learn for the GP kernel.

6.1 Baselines and competitors

Mkl-Svm.

This represents a standard, classifier driven approach to object localization, similar to [Vedaldi et al.(2009)Vedaldi, Gulshan, Varma, and Zisserman, Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders]. On of the training set we train a separate SVM for each group of features described above. We combine these classifiers by training a linear SVM over their outputs on the remaining of the data. We also include the location and scale features of a window in this second-level SVM. This baseline uses exactly the same candidate windows [Manen et al.(2013)Manen, Guillaumin, and Van Gool] and features as our method (sec. 6).

Uva [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders].

The popular method [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders] can be seen as a smaller version of the MKL-SVM baseline we have just defined. In order to make a more exact comparison to [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders], we remove the additional features and use only their three SIFT bag-of-words. Moreover, instead of training a second level SVM, we simply combine their outputs by averaging. This corresponds to [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders], but using the recent state-of-the-art object proposals [Manen et al.(2013)Manen, Guillaumin, and Van Gool] instead of selective search proposals. This method [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders] is one of the best performing object detectors. It has won the ILSVRC 2011 [ils(2011)] detection challenge and the PASCAL VOC 2012 detection challenge.

Ae-Gp [Vezhnevets and Ferrari(2014)].

Finally, we compare to the AE-GP+ model of [Vezhnevets and Ferrari(2014)]. It corresponds to a degenerate version of our model which uses only to score each window in isolation by looking at its own features (5). This uses the same candidate windows [Manen et al.(2013)Manen, Guillaumin, and Van Gool] and features we use. This technique was shown to outperform earlier work on location annotation in ImageNet [Guillaumin and Ferrari(2012)].

Figure 4: Mean overlap curves. We retain the single top scoring candidate window in a test image and measure the mean overlap of the output windows with the ground-truth. We vary a threshold on the score of the output windows to generate performance curves. The higher the curve, the better.

6.2 Results

For each method we retain the single top scoring candidate window in a test image. We measure localization accuracy as the mean overlap of the output windows with the ground-truth [Vezhnevets and Ferrari(2014)] (fig. 4). We vary a threshold on the score of the output windows to generate performance curves. The higher the curve, the better.

As fig. 4 shows, the proposed method consistently outperforms the competitors and the baseline over the whole range of the curve. Our method achieves 0.75 mean overlap when returning annotations for 35% of all images. At the same accuracy level, AE-GP, MKL-SVM and UVA return , and images respectively, i.e. we can return more annotation at this high level of overlap than the closest competitor. Producing very accurate bounding-box annotations is important for their intended use as training data for various models and tasks. Improving over AE-GP validates the proposed idea of scoring candidate windows by taking into account spatial relations to all other windows and their appearance similarity. The favourable comparison to the excellent method UVA [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders] and to its extended MKL-SVM version demonstrates that our system offers competitive performance. Example objects localized by our method and by AE-GP are shown in fig. 5. Our method successfully operates in cluttered images (guenon, barrow, skunk). It can find camouflaged animals (tiger), small objects (buckle, racket), and deal with diverse classes and high intra-class variation (pool cue, buckle, racket).

To evaluate the impact of our fast inference algorithm (sec. 4.1) we compared it to brute force (i.e. evaluating the energy for all possible configurations) on the baseball class. On average brute force takes 17.6s per image, whereas our fast inference takes 0.14s. Since our inference method is exact, it produces the same solution as brute force, but 124 faster.

Figure 5: Qualitative results. Results of our method (red) vs AE-GP [Vezhnevets and Ferrari(2014)] (green). Notice, how our method is able to detect small, off-center objects despite occlusion (pool cue) or the object blending with its surroundings (tiger).

6.3 Conclusion

We have presented a new method for annotating the location of objects in ImageNet, which goes beyond considering one candidate window at a time. Instead, it scores each window in the context of all other windows in the image, taking into account their similarity in appearance space as well as their spatial relations in the image plane. As we have demonstrated on 92K images from ImageNet, our method improves over some of the best performing object localization techniques [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders, Vezhnevets and Ferrari(2014)], including the one we build on [Vezhnevets and Ferrari(2014)].

Acknowledgment

This work was supported by the ERC Starting Grant VisCul. A. Vezhnevets was also supported by SNSF fellowship PBEZP-2142889.

References

  • [ils(2011)] Imagenet large scale visual recognition challenge (ILSVRC). http://www.image-net.org/challenges/LSVRC/2011/index, 2011.
  • [ils(2013)] University of amsterdam and euvision technologies at ilsvrc2013 (ILSVRC). http://www.image-net.org/challenges/LSVRC/2013/slides/ILSVRC2013-UvA-Euvision-web.pdf, 2013.
  • [Alexe et al.(2012)Alexe, Deselaers, and Ferrari] B. Alexe, T. Deselaers, and V. Ferrari. Measuring the objectness of image windows. IEEE Trans. on PAMI, 2012.
  • [Andriluka et al.(2010)Andriluka, Roth, and Schiele] M. Andriluka, S. Roth, and B. Schiele. Monocular 3d pose estimation and tracking by detection. In CVPR, 2010.
  • [Bakir et al.(2007)Bakir, Schölkopf, Smola, Taskar, Vishwanathan, and S.V.N] G.H. Bakir, B. Schölkopf, A. J. Smola, Ben Taskar, Vishwanathan, and S.V.N. Predicting structured data, 2007.
  • [Blaschko and Lampert(2008)] M. B. Blaschko and C. H. Lampert. Learning to localize objects with structured output regression. In ECCV, 2008.
  • [Choi et al.(2010)Choi, Lim, Torralba, and Willsky] M. Choi, J. Lim, A. Torralba, and A. Willsky. Exploiting hierarchical context on a large database of object categories. In CVPR, 2010.
  • [Dalal and Triggs(2005)] N. Dalal and B. Triggs. Histogram of Oriented Gradients for human detection. In CVPR, 2005.
  • [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-fei] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
  • [Desai et al.(2009)Desai, Ramanan, and Folkess] C Desai, D. Ramanan, and C. Folkess. Discriminative models for multi-class object layout. In ICCV, 2009.
  • [Divvala et al.(2009)Divvala, Hoiem, Hays, Efros, and Hebert] S. K. Divvala, D. Hoiem, J. H Hays, A. Efros, and M. Hebert. An empirical study of context in object detection. In CVPR, pages 1271–1278, 2009.
  • [Dollar and Zitnick(2014)] P. Dollar and C. Zitnick. Edge boxes: Locating object proposals from edges. In ECCV, 2014.
  • [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. IJCV, 2010.
  • [Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  • [Guillaumin and Ferrari(2012)] M. Guillaumin and V. Ferrari. Large-scale knowledge transfer for object localization in imagenet. In CVPR, 2012.
  • [Guillaumin et al.(2014)Guillaumin, Küttel, and Ferrari] M. Guillaumin, D. Küttel, and V Ferrari. ImageNet auto-annotation with segmentation propagation. IJCV, 2014.
  • [Harzallah et al.(2009)Harzallah, Jurie, and Schmid] H. Harzallah, F. Jurie, and C. Schmid. Combining efficient object localization and image classification. In ICCV, 2009.
  • [Heitz and Koller(2008)] G. Heitz and D. Koller. Learning spatial context: Using stuff to find things. In ECCV, 2008.
  • [Lampert et al.(2008)Lampert, Blaschko, and Hofmann] C. H. Lampert, M. B. Blaschko, and T. Hofmann. Beyond sliding windows: Object localization by efficient subwindow search. In CVPR, 2008.
  • [Lehmann et al.(2011)Lehmann, Gehler, and Van Gool] A. Lehmann, P. V Gehler, and L. J. Van Gool. Branch&rank: Non-linear object detection. In BMVC, volume 2, page 1, 2011.
  • [Leibe et al.(2007)Leibe, Schindler, and Van Gool] B. Leibe, K. Schindler, and L. Van Gool. Coupled detection and trajectory estimation for multi-object tracking. In ICCV, 2007.
  • [Li et al.(2011)Li, Parikh, and Chen] C. Li, D. Parikh, and T. Chen. Extracting adaptive contextual cues from unlabeled regions. In ICCV, pages 511–518. IEEE, 2011.
  • [Manen et al.(2013)Manen, Guillaumin, and Van Gool] S. Manen, M. Guillaumin, and L. Van Gool. Prime object proposals with randomized prim’s algorithm. In ICCV, 2013.
  • [Mottaghi et al.(2014)Mottaghi, Chen, Liu, Cho, Lee, Fidler, Urtasun, and Yuille] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891–898. IEEE, 2014.
  • [Murphy et al.(2003)Murphy, Torralba, and Freeman] K. Murphy, A. Torralba, and W. T. Freeman. Using the forest to see the trees: A graphical model relating features, objects, and scenes. In NIPS, 2003.
  • [Rabinovich et al.(2007)Rabinovich, Vedaldi, Galleguillos, Wiewiora, and Belongie] A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, and S. Belongie. Objects in context. In ICCV, 2007.
  • [Rasmussen and Williams(2005)] C. E. Rasmussen and C. K. I. Williams.

    Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

    .
    The MIT Press, 2005.
  • [Russell et al.(2007)Russell, Torralba, Liu, Ferugs, and Freeman] B. Russell, A. Torralba, C. Liu, R. Ferugs, and W. Freeman. Object recognition by scene alignment. In NIPS, 2007.
  • [Song et al.(2011)Song, Chen, Huang, Hua, and Yan] Z. Song, Q. Chen, Z. Huang, Y. Hua, and S. Yan. Contextualizing object detection and classification. In CVPR, 2011.
  • [Torralba(2003)] A. Torralba. Contextual priming for object detection. IJCV, 53(2):153–167, 2003.
  • [Tsochantaridis et al.(2005)Tsochantaridis, Joachims, Hofmann, and Altun] I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large margin methods for structured and interdependent output variables. JMLR, 6:1453–1484, 2005.
  • [Uijlings et al.(2013)Uijlings, van de Sande, Gevers, and Smeulders] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. Selective search for object recognition. IJCV, 2013.
  • [Vedaldi and Zisserman(2009)] A. Vedaldi and A. Zisserman. Structured output regression for detection with partial truncation. In NIPS, 2009.
  • [Vedaldi et al.(2009)Vedaldi, Gulshan, Varma, and Zisserman] A. Vedaldi, V. Gulshan, M. Varma, and A. Zisserman. Multiple kernels for object detection. In ICCV, 2009.
  • [Vezhnevets and Ferrari(2014)] A. Vezhnevets and V. Ferrari. Associative embeddings for large-scale knowledge transfer with self-assessment. In CVPR, 2014.
  • [Wang et al.(2013)Wang, Yang, Zhu, and Lin] X. Wang, M. Yang, S. Zhu, and Y. Lin. Regionlets for generic object detection. In ICCV, pages 17–24. IEEE, 2013.