The toolbox for the Google Refexp dataset proposed in this paper: http://arxiv.org/abs/1511.02283
We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MS-COCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/mjhucla/Google_Refexp_toolboxREAD FULL TEXT VIEW PDF
The toolbox for the Google Refexp dataset proposed in this paper: http://arxiv.org/abs/1511.02283
There has been a lot of recent interest in generating text descriptions of images (see e.g., [13, 53, 9, 5, 12, 26, 28, 40, 55, 8]). However, fundamentally this problem of image captioning is subjective and ill-posed. With so many valid ways to describe any given image, automatic captioning methods are thus notoriously difficult to evaluate. In particular, how can we decide that one sentence is a better description of an image than another?
In this paper, we focus on a special case of text generation given images, where the goal is to generate an unambiguous text description that applies to exactly one object or region in the image. Such a description is known as a “referring expression”[50, 52, 41, 42, 14, 19, 27]. This approach has a major advantage over generic image captioning, since there is a well-defined performance metric: a referring expression is considered to be good if it uniquely describes the relevant object or region within its context, such that a listener can comprehend the description and then recover the location of the original object. In addition, because of the discriminative nature of the task, referring expressions tend to be more detailed (and therefore more useful) than image captions. Finally, it is easier to collect training data to “cover” the space of reasonable referring expressions for a given object than it is for a whole image.
We consider two problems: (1) description generation, in which we must generate a text expression that uniquely pinpoints a highlighted object/region in the image and (2) description comprehension, in which we must automatically select an object given a text expression that refers to this object (see Figure 1). Most prior work in the literature has focused exclusively on description generation (e.g., [31, 27]). Golland et al.  consider generation and comprehension, but they do not process real world images.
In this paper, we jointly model both tasks of description generation and comprehension, using state-of-the-art deep learning approaches to handle real images and text. Specifically, our model is based upon recently developed methods that combine convolutional neural networks (CNNs) with recurrent neural networks (RNNs). We demonstrate that our model outperforms a baseline which generates referring expressions without regard to the listener who must comprehend the expression. We also show that our model can be trained in a semi-supervised fashion, by automatically generating descriptions for image regions.
Being able to generate and comprehend object descriptions is critical in a number of applications that use natural language interfaces, such as controlling a robot (e.g., “Rosie, please fetch me the beer from the top shelf of the fridge”, cf. ), or interacting with photo editing software (e.g., “Picasa, please replace the third car behind the fence with a motorbike”, cf. ). In addition, it is a good test bed for performing research in the area of vision and language systems because of the existence of a useful objective performance measure.
In order to train and evaluate our system, we have collected and released a new large scale referring expressions dataset based on the popular MS-COCO dataset .
To summarize, our main contributions are as follows. First, we present a new large scale dataset for referring expressions. Second, we evaluate how existing image captioning methods perform at the referring expression task. Third, we develop a new method for joint generation and comprehension that outperforms current methods.
Referring expressions. Referring expression generation is a classic NLP problem (see e.g., [54, 31]). Important issues include understanding what types of attributes people typically use to describe visual objects (such as color and size) , usage of higher-order relationships (e.g., spatial comparison) 
, and the phenomena of over and under-specification, which is also related to speaker variance.
Context (sometimes called pragmatics ) plays a critical role in several ways . First, the speaker must differentiate the target object from a collection of alternatives and must thus reason about how the object differs from its context. Second, the perception of the listener is also valuable. In particular, Golland et al.  recently proposed a game theoretic formulation of the referring expression problem showing that speakers that act optimally with respect to an explicit listener model naturally adhere to the Gricean Maxims of communication .
In most of this previous work, authors have focused on small datasets of computer generated objects (or photographs of simple objects) [50, 41] and have not connected their text generation systems to real vision systems. However there has been recent interest in understanding referring expressions in the context of complex real world images, for which humans tend to generate longer phrases . Kazemzadeh et al.  were the first to collect a large scale dataset of referring expressions for complex real world photos.
We likewise collect and evaluate against a large scale dataset. However we go beyond expression generation and jointly learn both generation and comprehension models. And where prior works have had to explicitly enumerate attribute categories such as size, color (e.g. ) or manually list all possible visual phrases (e.g. ), our deep learning-based models are able to learn to directly generate surface expressions from raw images without having to first convert to a formal object/attribute representation.
Concurrently,  propose a CNN-RNN based method that is similar to our baseline model and achieve state-of-the-art results on the ReferIt dataset . But they did not use the discriminative training strategy proposed in our full model. [25, 32] investigate the task of generating dense descriptions in an image. But their descriptions are not required to be unambiguous.
Image captioning. Our methods are inspired by a long line of inquiry in joint models of images and text, primarily in the vision and learning communities [13, 23, 49, 43, 34, 56, 36]. From a modeling perspective, our approach is closest to recent works applying RNNs and CNNs to this problem domain [53, 9, 5, 12, 26, 28, 40, 55]. The main approach in these papers is to represent the image content using the hidden activations of a CNN, and then to feed this as input to an RNN, which is trained to generate a sequence of words.
Most papers on image captioning have focused on describing the full image, without any spatial localization. However, we are aware of two exceptions. 
propose an attention model which is able to associate words to spatial regions within an image; however, they still focus on the full image captioning task. propose a model for aligning words and short phrases within sentences to bounding boxes; they then train an model to generate these short snippets given features of the bounding box. Their model is similar to our baseline model, described in Section 5 (except we provide the alignment of phrases to boxes in the training set, similar to ). However, we show that this approach is not as good as our full model, which takes into account other potentially confusing regions in the image.
Visual question answering. Referring expressions is related to the task of VQA (see e.g., [2, 38, 39, 16, 15]). In particular, referring expression comprehension can be turned into a VQA task where the speaker asks a question such as “where in the image is the car in red?” and the system must return a bounding box (so the answer is numerical, not linguistic). However there are philosophical and practical differences between the two tasks. A referring expression (and language in general) is about communication — in our problem, the speaker is finding the optimal way to communicate to the listener, whereas VQA work typically focuses only on answering questions without regard to the listener’s state of mind. Additionally, since questions tend to be more open ended in VQA, evaluating their answers can be as hard as with general image captioning, whereas evaluating the accuracy of a bounding box is easy.
The largest existing referring expressions dataset that we know of is the ReferIt dataset, which was collected by , and contains 130,525 expressions, referring to 96,654 distinct objects, in 19,894 photographs of natural scenes. Images in this dataset are from the segmented and annotated TC-12 expansion of the ImageCLEF IAPR dataset . Two drawbacks of this dataset, however, are that (1) the images sometimes only contain one object of a given class, allowing speakers to use short descriptions without risking ambiguity, and (2) the ImageCLEF dataset focuses mostly on “stuff” (i.e. context) rather than “things” (i.e. objects).
In this paper, we use a similar methodology to that of , but building instead on top of the MSCOCO dataset , which contains more than 300,000 images, with 80 categories of objects segmented at the instance level.
For each image, we selected objects if (1) there are between 2 and 4 instances of the same object type within the same image, and (2) if their bounding boxes occupy at least 5% of image area. This resulted in selecting 54,822 objects from 26,711 images. We constructed a Mechanical Turk task in which we presented each object in each image (by highlighting the object mask) to a worker whose task was to generate a unique text description of this object. We then used a second task in which a different worker was presented with the image and description, and was asked to click inside the object being referred to. If the selected point was inside the original object’s segmentation mask, we considered the description as valid, and kept it, otherwise we discarded it and re-annotated it by another worker. We repeated these description generation and verification tasks on Mechanical Turk iteratively up to three times. In this way, we selected 104,560 expressions. Each object has on average 1.91 expressions, and each image has on average 3.91 expressions. This dataset is denoted as Google Refexp dataset and some samples are shown in Figure 2. We have released this dataset and a toolbox for visualization and evaluation, see https://github.com/mjhucla/Google_Refexp_toolbox.
While we were collecting our dataset, we learned that Tamara Berg had independently applied her ReferIt game  to the MSCOCO dataset to generate expressions for 50,000 objects from 19,994 images. She kindly shared her data (named as UNC-Ref-COCO dataset) with us. For brevity, we call our Google Refexp dataset as G-Ref and the UNC-Ref-COCO as UNC-ref. We report results on both datasets in this paper. However, due to differences in our collection methodologies, we have found that the descriptions in the two overlapped datasets exhibit significant qualitative differences, with descriptions in the UNC-Ref dataset tending to be more concise and to contain less flowery language than our descriptions. 222According to our personal communication with the authors of the UNC-Ref dataset, the instruction and reward rule of UNC-Ref encourages the annotators to give a concise description in a limited time, while in our G-Ref dataset, we encourage the annotators to give rich and natural descriptions. This leads to different styles of annotations. More specifically, the average lengths of expressions from our dataset and UNC-Ref are 8.43 and 3.61 respectively. And the size of the word dictionaries (keeping only words appearing more than 3 times) from our dataset and UNC-Ref are 4849 and 2890 respectively. See Figure 3 for some visual comparisons.
In this section, we describe at a high level how we solve the two main tasks of description and generation. We will describe the model details and training in the next section.
In the description generation task, the system is given a full image and a target object (specified via a bounding box), and it must generate a referring expression for the target object. Formally, the task is to compute , where is a sentence, is a region, and is an image.
Since we will use RNNs to represent , we can generate
one word at a time until we generate an end of sentence symbol. Computing the globally most probable sentence is hard, but we can use beam search to approximately find the most probable sentences (we use a beam size of 3). This is very similar to a standard image captioning task, except the input is a region instead of a full image. The main difference is that we will train our model to generate descriptions that distinguish the input region from other candidate regions.
In the description comprehension task, we are given a full image and a referring expression and are asked to localize the the object being referred to within the image by returning a bounding box. One approach would be to train a model to directly predict the bounding box location given the referring expression (and image). However, in this paper, we adopt a simpler, ranking-based approach. In particular, we first generate a set of region proposals, and then ask the system to rank these by probability. Then we select the region using , where, by Bayes’ rule, we have
If we assume a uniform prior for ,333 This implies that we are equally likely to choose any region to describe. This is approximately true by virtue of the way we constructed the dataset. However, in real applications, region saliency should be taken into account. we can select the region using
. This strategy is similar to image retrieval methods such as[29, 40], where the regions play the role of images.
At test time, we use the multibox method of 
to generate objects proposals. This generates a large number of class agnostic bounding boxes. We then classify each box into one of the 80 MS-COCO categories, and discard those with low scores. We use the resulting post-classification boxes as the proposal set. To get an upper bound on performance, we also use the ground truth bounding boxes for all the objects in the image. In both cases, we do not use the label for the object of interest when ranking proposals.
In this section we explain our baseline method for computing .
Our baseline model is similar to other image captioning models that use a CNN to represent the image, followed by an LSTM to generate the text (see e.g., [40, 9, 53]). The main difference is that we augment the CNN representation of the whole image with a CNN representation of the region of interest, in addition to location information. See Figure 4 for an illustration of our baseline model.
In more detail, we use VGGNet 
as our CNN, pre-trained on the ImageNet dataset[7, 33]. The last 1000 dimensional layer of VGGNet is used as our representation of the object region. In addition, we compute features for the whole image, to serve as context. In experiments, we only fine-tuned the weights for the last layer of the CNN and fixed all other layers. To feed a region to the CNN, we keep the aspect ratio of the region fixed and scale it to
resolution, padding the margins with the mean pixel value (this is similar to the region warping strategy in
). This gives us a 2000-dimensional feature vector, for the region and image.
We encode the relative location and size of the region using a 5 dimensional vector as follows: , where and are the coordinates of the top left and bottom right corners of the object bounding box, and are height and width of the image, and and are the sizes of the bounding box and image respectively.
Concatenating with the region, image, and location/size features, we obtain a 2005-dimensional vector which we feed as input into an LSTM sequence model, which parameterizes the form of the distribution . For our LSTMs, we use a 1024-dimensional word-embedding space, and 1024-dimensional hidden state vector. We adopt the most commonly used vanilla LSTM structure  and feed the visual representation as input to the LSTM at each time step.
Our training data (discussed in Section 3) consists of observed triplets , where is an image, denotes a region within , and denotes a referring expression for . To train the baseline model, we minimize the negative log probability of the referring expressions given their respective region and image:
where are the parameters of the RNN and CNN, and where we sum over the examples in the training set. We use ordinary stochastic gradient decent with a batch size of 16 and use an initial learning rate of 0.01 which is halved every 50,000 iterations. Gradient norms are clipped to a maximum value of 10. To combat overfitting, we regularize using dropout with a ratio of 0.5 for both the word-embedding and output layers of the LSTM.
The baseline method is to train the model to maximize , as is common for CNN-LSTM based image captioning models. However a strategy that directly generates an expression based only on the target object (which  calls the reflex speaker strategy) has the drawback that it may fail to generate discriminative sentences. For example, consider Figure 4: to generate a description of the girl highlighted by the green bounding box, generating the word “pink” is useful since it distinguishes this girl from the other girl on the right. To this end, we propose a modified training objective, described below.
Section 5.2 proposed a way to train the model using maximum likelihood. We now propose the following alternative objective function:
We will call this the softmax loss. Note that this is the same as maximizing the mutual information between and (assuming a uniform prior for ), since
where . Hence this approach is also called Maximum Mutual Information (MMI) training .
The main intuition behind MMI training is that we want to consider whether a listener would interpret the sentence unambiguously. We do this by penalizing the model if it thinks that a referring expression for a target object could also be plausibly generated by some other object within the same image. Thus given a training sample , we train a model that outputs a high , while maintaining a low , whenever . Note that this stands in contrast to the Maximum Likelihood (ML) objective function in Equation 2 which directly maximizes without considering other objects in the image.
There are several ways to select the region proposals . We could use all the true object bounding boxes, but this tends to waste time on objects that are visually very easy to discriminate from the target object (hence we call these “easy ground truth negatives”). An alternative is to select true object bounding boxes belonging to objects of the same class as the target object; these are more confusable (hence we call them “hard ground truth negatives”). Finally, we can use multibox proposals, the same as we use at test time, and select the ones with the same predicted object labels as (hence we call them “hard multibox negatives”). We will compare these different methods in Section 8.2. We use 5 random negatives at each step, so that all the data for a given image fits into GPU memory.
To optimize Equation 3, we must replicate the network (using tied weights) for each region (including the true region ), as shown in Figure 5. The resulting MMI trained model has exactly the same number of parameters as the ML trained model, and we use the same optimization and regularization strategy as in Section 5.2. Thus the only difference is the objective function.
For computational reasons, it is more convenient to use the following max-margin loss, which compares the target region against a single random negative region :
This objective, which we call max-margin MMI (or MMI-MM) intuitively captures a similar effect as its softmax counterpart (MMI-SoftMax) and as we show in Section 8.2, yields similar results in practice. However, since the max-margin objective only compares two regions, the network must only be replicated twice. Consequently, less memory is used per sentence, allowing for more sentences to be loaded per minibatch which in turn helps in stabilizing the gradient.
Collecting referring expressions data can be expensive. In this section we discuss semi-supervised training of our full model by making use of bounding boxes that do not have descriptions, and thus are more ubiquitously available. Our main intuition for why a bounding box (region) can be useful even without an accompanying description is because it allows us to penalize our model during MMI training if it generates a sentence that it cannot itself decode to correctly recover (recall that MMI encourages to be higher than , whenever ).
In this semi-supervised setting, we consider a small dataset of images with bounding boxes and descriptions, together with a larger dataset of images and bounding boxes, but without descriptions. We use to train a model (which we call model ) to compute . We then use this model to generate a set of descriptions for the bounding boxes in (we call this new dataset ). We then retrain on , in the spirit of bootstrap learning.
The above strategy suffers from the flaw that not all of the generated sentences are reliable, which may “pollute” the training set. To handle this, we train an ensemble of different models on (call them model ), and use these to determine which of the generated sentences for are trustworthy. In particular, we apply each model in the ensemble to decode each sentence in , and only keep the sentence if every model maps it to the same correct object; we will call the resulting verified dataset . This ensures that the generator creates referring expressions that can be understood by a variety of different models, thus minimizing overfitting. See Figure 6 for an illustration. In the experiments, we show that our model benefits from this semi-supervised training.
We conducted experiments on both of the COCO referring expression datasets mentioned in Section 3: our G-Ref dataset and the UNC-Ref dataset. We randomly chose 5,000 objects as the validation set, 5,000 objects as the testing set and the remaining objects as the training set (44,822 for G-Ref and 40,000 for UNC-Ref).
In this section, we describe how we evaluate performance of the comprehension and generation tasks.
The comprehension task is easy to evaluate: we simply compute the Intersection over Union (IoU) ratio between the true and predicted bounding box. If IoU exceeds 0.5, we call the detection a true positive, otherwise it is a false positive (this is equivalent to computing the precision measure). We then average this score over all images.
The generation task is more difficult — we can evaluate a generated description in the same way as an image description, using metrics such as CIDEr , BLEU  and METEOR . However these metrics can be unreliable and do not account for semantic meaning. We rely instead on human evaluation, as was done in the most recent image captioning competition . In particular, we asked Amazon Mechanical Turk (AMT) workers to compare an automatically generated object description to a human generated object description, when presented with an image and object of interest. The AMT workers do not know which sentences are human generated and which are computer generated (we do not even tell them that some sentences might be computer generated to reduce possible bias). We simply ask them to judge which sentence is a better description, or if they are equally good.
In addition to human evaluation, which does not scale, we evaluate our entire system by passing automatically generated descriptions to our comprehension system, and verifying that they get correctly decoded to the original object of interest. This end-to-end test is automatic and much more reliable than standard image captioning metrics.
In this section, we compare different ways of training our model: maximum likelihood training (the baseline method); max-margin loss with easy ground truth negatives (“MMI-MM-easy-GT-neg”); max-margin loss with hard ground truth negatives (“MMI-MM-hard-GT-neg”); max-margin loss with hard multibox negatives (“MMI-MM-multibox-neg”); softmax/MMI loss with hard multibox negatives (“MMI-SoftMax”). For each method, we consider using either ground truth or multibox proposals at test time. In addition, we consider both ground truth descriptions and generated descriptions.
In this experiment we treat UNC-Ref as a validation set to explore various algorithmic options and hyperparameter settings for MMI. Only after having fixed these algorithmic options and hyperparameter settings did we do experiments on our G-Ref dataset (Section8.3). This reduces the risk that we will have “overfit” our hyperparameters to each particular dataset. The results are summarized in Table 1 and we draw the following conclusions:
All models perform better on generated descriptions than the groundtruth ones, possibly because the generated descriptions are shorter than the groundtruth (5.99 words on average vs 8.43), and/or because the generation and comprehension models share the same parameters, so that even if the generator uses a word incorrectly (e.g., describing a “dog” as a “cat”), the comprehension system can still decode it correctly. Intuitively, a model might “communicate” better with itself using its own language than with others.
All the variants of the Full model (using MMI training) work better than the strong baseline using maximum likelihood training.
The softmax version of MMI training is similar to the max-margin method, but slightly worse.
MMI training benefits more from hard negatives than easy ones.
Training on ground truth negatives helps when using ground truth proposals, but when using multibox proposals (which is what we can use in practice), it is better to use multibox negatives.
Based on the above results, for the rest of the paper we will use max-margin training with hard multibox negatives as our Full Model.
In this section, we compare the strong baseline (maximum likelihood) with our max-margin MMI method on the validation and test sets from G-Ref and UNC-Ref. As before, we consider ground truth and multibox proposals at test time, and ground truth (human) or generated (automatic) descriptions. The results are shown in Table 2. We see that MMI training outperforms ML training under every setting. 444We also train our baseline and full model on a random train, val, and test split w.r.t. to the images of our G-Ref dataset. The results are consistent with those in Table 2. With multibox proposals and GT descriptions, the Precision of the baseline and full model are 0.404 and 0.444 on val set, and 0.407 and 0.451 on test set respectively.
In addition to the above end-to-end evaluation, we use human evaluators to judge generated sentence quality. In particular, we selected 1000 objects at random from our test set, and showed them to Amazon Mechanical Turk workers. The percentage of descriptions that are evaluated as better or equal to a human caption for the baseline and the full model are 15.9% and 20.4% respectively. This shows that MMI training is much better (4.5% absolute improvement, and 28.5% relative) than ML training.
To conduct the semi-supervised training experiment, we separate the training set of our G-Ref dataset and the UNC-Ref dataset into two parts with the same number of objects. The first part (denoted by ) has the object description annotations while the second part (denoted by ) only has object bounding boxes. Table 3 shows the results of semi-supervised training on the validation set of our dataset and UNC-Ref. We see that we get some improvement by training on over just using .
In Figure 7 we show qualitative results of our full generation model (above the dashed line) and the baseline generation model (below the dashed line) on some of our test images. We see that the descriptions generated by our full model are typically longer and more discriminative than the baseline model. In the second image, for example, the baseline describes one of the cats as “a cat laying on a bed”, which is not sufficiently unambiguous for a listener to understand which cat is being described. Our full model, on the other hand, describes the same cat as “a cat laying on the left” which is completely unambiguous.
Figure 8 shows some qualitative results of our full comprehension model on our test dataset. The first and second columns show the original image and the multibox proposals respectively. The last four columns show the bounding boxes (denoted as a red bounding box in the figure) selected by our full model in response to different input sentences (both ground truth sentences and ones we created to probe the comprehension abilities of the model). To better interpret these results, we also show the bounding boxes that are within the margin of the model (see Eqn. 6) with dashed blue bounding boxes. Their bounding boxes are considered as “possible candidates” but their scores (i.e. ) are not as high as the chosen one.
In general, we see that the comprehension model does quite well from short two word phrases to longer descriptions. It is able to respond correctly to single word changes in a referring expression (e.g., “the man in black” to “the man in red”). It also correctly identifies that the horse is the referent of the expression “a dark horse carrying a woman” whereas the woman is the referent in “a woman on the dark horse” — note that methods that average word embeddings would most likely fail on this example. However, there are also failure cases. For example, in the fifth row, “the woman in white” selects a woman in black; this is because our model cannot handle the case where the object is not present, although it makes a reasonable guess. Also, in the fifth row, “the controller in the woman’s hand” selects the woman, the orange juice and the controller, since this particular kind of object is too small to detect, and lacks enough training data.
To conclude, we leave the reader with two simple points. First, referring expressions have been studied for decades, but in light of the recent burst of interest in image captioning, referring expressions take on new importance. Where image captioning itself is difficult to evaluate, referring expressions have an objective performance metric, and require the same semantic understanding of language and vision. Thus success on datasets such as the one contributed in this paper is more meaningful than success by standard image captioning metrics.
Second, to be successful at generating descriptions, we must consider the listener. Our experiments show that modeling a listener that must correctly decode a generated description consistently outperforms a model that simply emits captions based on region features. We hope that in addition to our dataset, these insights will spur further progress on joint models of vision and language.
We are grateful to Tamara Berg for sharing the UNC-Ref-COCO dataset. We also thank Sergio Guadarrama, Vivek Rathod, Vignesh Ramanathan, Nando de Freitas, Rahul Sukthankar, Oriol Vinyals and Samy Bengio for early discussions and feedback on drafts. This work was partly supported by ARO 62250-CS, the NSF Center for Brains, Minds, and Machines, and NSF STC award CCF-1231216.
Scalable object detection using deep neural networks.In CVPR, pages 2155–2162, 2014.
Visual turing test for computer vision systems.PANS, 112(12):3618–3623, 2015.
Framing image description as a ranking task: Data, models and evaluation metrics.JAIR, 47:853–899, 2013.
Composing simple image descriptions using web-scale n-grams.In CoNLL, pages 220–228, 2011.
Ask your neurons: A neural-based approach to answering questions about images.In NIPS, 2015.