Multilabel image annotation [25, 14] is an important and challenging problem in computer vision. Most existing work focus on single-label classification problems [6, 21], where each image is assumed to have only one class label. However, this is not necessarily true for real world applications, as an image may be associated with multiple semantic tags (Figure 1). As a practical example, images from Flickr are usually accompanied by several tags to describe the content of the image, such as objects, activities, and scene descriptions. Images on the Internet, in general, are usually associated with sentences or descriptions, instead of a single class label, which may be deemed as a type of multitagging. Therefore, it is a practical and important problem to accurately assign multiple labels to one image.
Single-label image classification has been extensively studied in the vision community, the most recent advances reported on the large-scale ImageNet database. Most existing work focus on designing visual features for improving recognition accuracy. For example, sparse coding [33, 36]
, Fisher vectors, and VLAD  have been proposed to reduce the quantization error of “bag of words”-type features. Spatial pyramid matching 
has been developed to encode spatial information for recognition. Very recently, deep convolutional neural networks (CNN) have demonstrated promising results for single-label image classification. Such algorithms have all focused on learning a better feature representation for one-vs-rest classification problems, and it is not yet clear how to best train an architecture for multilabel annotation problems.
In this work, we are interested in leveraging the highly expressive convolutional network for the problem of multilabel image annotation. We employed a similar network structure to , which contains several convolutional and dense connected layers as the basic architecture. We studied and compared several other popular multilabel losses, such as the ranking loss  that optimizes the area under ROC curve (AUC), and the cross-entropy loss used in Tagprop . Specifically, we propose to use the top- ranking loss, inspired by , for embedding to train the network. Using the largest publicly available multilabel dataset NUS-WIDE , we observe a significant performance boost over conventional features, reporting the best retrieval performance on the benchmark dataset in the literature.
1.1 Previous Work
In this section we first review related works on multilabel image annotation and then briefly discuss works on deep convolutional networks.
Modeling Internet images and their corresponding textural information (e.g., sentences, tags) have been of great interest in the vision community [2, 10, 12, 15, 30, 29, 34]. In this work, we focus on the image annotation problem and summarize several important lines of related research. Early work in this area was mostly devoted to annotation models inspired by machine translation techniques [1, 8]. The work by Barnard et al. [1, 8] applied machine translation methods to parse natural images and tried to establish a relationship between image regions and words.
, centred upon the idea of learning a parametric model to perform predictions. However, because image annotation is a highly nonlinear problem, a parametric model might not be sufficient to capture the complex distribution of the data. Several recent works on image tagging have mostly focused on nonparametric nearest-neighbor methods, which offer higher expressive power. The work by Makadia et al., which proposed a simple nearest-neighbor-based tag transfer approach, achieved significant improvement over previous model-based methods. Recent improvements on the nonparametric approach include TagProp , which learns a discriminative metric for nearest neighbors to improve tagging.
Convolutional neural networks (CNNs) [20, 22, 23, 17, 5] are a special type of neural network that utilizes specific network structures, such as convolutions and spatial pooling, and have exhibited good generalization power in image-related applications. Combined with recent techniques such as Dropout and fast parallel training, CNN models have outperformed existing hancrafted features. Krizhevsky et al.  reported record-breaking results on ILSVRC 2012 that contains 1000 visual-object categories. However, this study was mostly concerned with single-label image classification, and the images in the dataset only contain one prominent object class. At a finer scale, several methods focus on improving specific network designs. Notably, Zeiler et al.  investigated different pooling methods for training CNNs, and several different regularization methods, such as Dropout , DropConnect , and Maxout  have been proposed to improve the robustness and representation power of the networks. In adition, Earlier studies  have shown that CNN features are suitable as a general feature for various tasks under the conventional classification schemes, and our work focuses on how to directly train a deep network from raw pixels, using multilabel ranking loss, to address the multilabel annotation problem.
2 Multilabel Deep Convolutional Ranking Net
In our approach for multilabel image annotation, we adopted the architecture proposed in 
as our basic framework and mainly focused on training the network with loss functions tailored for multi-label prediction tasks.
2.1 Network Architecture
The basic architecture of the network we use is similar to the one used in . We use five convolutional layers and three densely connected layers. Before feeding the images to the convolutional layers, each image is resized to 256256. Next, 220
220 patches are extracted from the whole image, at the center and the four corners to provide an augmentation of the dataset. Convolution filter sizes are set to squares of size 11, 9, and 5 respectively for the different convolutional layers; and max pooling layers are used in some of the convolutional layers to introduce invariance. Each densely connected layer has output sizes of 4096. Dropout layers follow each of the densely connected layers with a dropout ratio of 0.6. For all the layers, we used rectified linear units (RELU) as our nonlinear activation function.
The optimization of the whole network is achieved by asynchronized stochastic gradient descent with a momentum term with weight, with mini-batch size of 32. The global learning rate for the whole network is set to
at the beginning, and a staircase weight decay is applied after a few epochs. The same optimization parameters and procedure are applied to all the different methods. For our dataset with 150,000 training images, it usually takes one day to obtain a good model by training on a cluster. Unlike previous work that usually used ImageNet to pre-train the network, we train the whole network directly from the training images from the NUS-WIDE dataset for a fair comparison with conventional vision baselines.
2.2 Multilabel Ranking Losses
We mainly focused on loss layer, which specifies how the network training penalizes the deviation between the predicted and true labels, and investigated several different multilabel loss functions for training the network. The first loss function was inspired by Tagprop , for which we minimized the multilabel softmax regression loss. The second loss was a simple modification of a pairwise-ranking loss , which takes multiple labels into account. The third loss function was a multilabel variant of the WARP loss , which uses a sampling trick to optimize top- annotation accuracy.
For notations, assume that we have a set of images and that we denote the convolutional network by where the convolutional layers and dense connected layers filter the images. The output of is a scoring function of the data point , that produces a vector of activations. We assume there are image training data and tags.
; therefore, we adopted it in our context. The posterior probability of an imageand class can be expressed as
where means the activation value for image and class
. We then minimized the KL-Divergence between the predictions and the ground-truth probabilities. Assuming that each image has multiple labels, and that we can form a label vectorwhere means the presence of a label and means absence of a label for an image, we can obtain ground-truth probability by normalizing as . If the ground truth probability for image and class is defined as , the cost function to be minimized is
where denotes the number of positive labels for each image. For the ease of exposition and without loss of generality, we set to be the same for all images.
2.2.2 Pairwise Ranking
The second loss function we considered was the pairwise-ranking loss , which directly models the annotation problem. In particular, we wanted to rank the positive labels to always have higher scores than negative labels, which led to the following minimization problem:
where is the positive labels and is the negative labels. During the back-propagation, we computed the sub-gradient of this loss function. One limitation of this loss is that it optimizes the area under the ROC curve (AUC) but does not directly optimize the top- annotation accuracy. Because for image annotation problems we were mostly interested in top- annotations, this pairwise ranking loss did not best fit our purpose.
2.2.3 Weighted Approximate Ranking (WARP)
The third loss we considered was the weighted approximate ranking (WARP), which was first described in . It specifically optimizes the top- accuracy for annotation by using a stochastic sampling approach. Such an approach fits the stochastic optimization framework of the deep architecture very well. It minimizes
where is a weighting function for different ranks, and is the rank for the th class for image . The weighting function used in our work is defined as:
We defined the as equal to , which is the same as . The weights defined by control the top- of the optimization. In particular, if a positive label is ranked top in the label list, then will assign a small weight to the loss and will not cost the loss too much. However, if a positive label is not ranked top,
will assign a much larger weight to the loss, which pushes the positive label to the top. The last question was how to estimate the rank. We followed the sampling method in : for a positive label, we continued to sample negative labels until we found a violation; then we recorded the number of trials s we sampled for negative labels. The rank was estimated by the following formulation
for classes and sampling trials. We computed the sub-gradient for this layer during optimization.
As a minor noite, the approximate objective we optimize is a looser upper bound compared to the original WARP loss proposed in . To see this, notice that in the original paper, it is assumed that the probability of sampling a violator is with a positive example with rank , where is the number of labels. Thus, there are labels with higher scores than . This is true only if all these labels are negative. However, in our case, since there may be other positive labels having higher scores than due to the multi-label nature of the problem, we effectively have .
3 Visual Feature based Image Annotation Baslines
We used a set of 9 different visual features and combined them to serve as our baseline features. Although such a set of features might not have been the best possible ones we could obtain, they already serve as a very strong visual representation, and the computation of such features is nontrivial. On top of these features, we ran two simple but powerful classifiers (kNN and SVM) for image annotation. We also experimented with Tagprop, but found it cannot easily scale to a large training set because of the time complexity. After using a small training set to train the Tagprop model, we found the performance to be unsatisfactory and therefore do not compare it here.
3.1 Visual Features
GIST : We resized each image to 200200 and used three different scales [8, 8, 4] to filter each RGB channel, resulting in 960-dimensional (3203) GIST feature vectors.
SIFT: We used two different sampling methods and three different local descriptors to extract texture features, which gave us a total of 6 different features. We used dense sampling and a Harris corner detector as our patch-sampling methods. For local descriptors, we extracted SIFT , CSIFT , and RGBSIFT , and formed a codebook of size 1000 using kmeans clustering; then built a two-level spatial pyramid  that resulted in a 5000-dimensional vector for each image. We will refer to these six features as D-SIFT, D-CSIFT, D-RGBSIFT, H-SIFT, H-CSIFT, and H-RGBSIFT.
HOG: To represent texture information at a larger scale, we used 22 overlapping HOG as described in . We quantized the HOG features to a codebook of size 1000 and used the same spatial pyramid scheme as above, which resulted in 5000-dimensional feature vectors.
Color: We used a joint RGB color histogram of 8 bins per dimension, for a 512-dimensional feature.
The same set of features were used in 
, and achieved state-of-the-art performance for image retrieval and annotation. The combination of this set of features has a total dimensionality of 36,472, which makes learning very expensive. We followed to perform simple dimensionality reductions to reduce computation. In particular, we performed a kernel PCA (KPCA) separately on each feature to reduce the dimensionality to 500. Then we concatenated all of the feature vectors to form a 4500-dimensional global image feature vector and performed different learning algorithms on it.
3.2 Visual feature + Nn
The simplest baseline that remains very powerful involves directly applying a weighted kNN on the visual feature vectors. NN is a very strong baseline for image annotation, as suggested by Makadia et al. , mainly because multilabel image annotation is a highly nonlinear problem and handling the heavily tailed label distribution is usually very difficult. By contrast, kNN is a highly nonlinear and adaptive algorithm that better handles rare tags. For each test image, we found its nearest neighbors in the training set and computed the posterior probability as
where indexes the labels of training data, when there is one label for this image, and when there is no label for this image. is the bandwidth that needs to be tuned. After obtaining the prediction probabilities for each image, we sorted the scores and annotated each testing image with the top- tags.
3.3 Visual feature + SVM
Another way to perform image annotation is to treat each tag separately and to train different one-vs-all classifiers. We trained a linear SVM  for each tag and used the output of the different SVMs to rank the tags. Because we had already performed nonlinear mapping to the data during the KPCA stage, we found a linear SVM to be sufficient. Thus we assigned top- tags to one image, based on the ranking of the output scores of the SVMs.
We performed experiments on the largest publicly available multilabel dataset, NUS-WIDE . This dataset contains 269,648 images downloaded from Flickr that have been manually annotated, with several tags (2-5 on average) per image. After ignoring the small subset of the images that are not annotated by any tag, we had a total of 209,347 images for training and testing. We used a subset of 150,000 images for training and used the rest of the images for testing. The tag dictionary for the images contains 81 different tags. Some sample images and annotations are shown in Figure 1.
4.2 Evaluation Protocols
We followed previous research  in our use of the following protocols to evaluate different methods. For each image, we assigned (e.g., ) highest-ranked tags to the image and compared the assigned tags to the ground-truth tags. We computed the recall and precision for each tag separately, and then report the mean-per-class recall and mean-per-class precision:
where is the number of tags, is the number of correctly annotated image for tag , is the number of ground-truth tags for tag , and is the number of predictions for tag . The above evaluations are biased toward infrequent tags, because making them correct would have a very significant impact on the final accuracy. Therefore we also report the overall recall and overall precision:
For the above two metrics, the frequent classes will be dominant and have a larger impact on final performance. Finally, we also report the percentage of recalled tag words out of all tag words as N+. We believe that evaluating all of these metrics makes the evaluation unbiased.
4.3 Baseline Parameters
In our preliminary evaluation, we optimized the parameters for the visual-feature-based baseline systems. For visual-feature dimensionality reduction, we followed the suggestions in Gong et al.  to reduce the dimensionality of each feature to 500 and then concatenated the PCA-reduced vectors into a 4500-dimensional global image descriptor, which worked as well as the original feature. For NN, we set the bandwidth to 1 and to 50, having found that these settings work best. For SVM, we set the regularization parameter to , which works best for this dataset.
|method / metric||per-class recall||per-class precision||overall recall||overall precision|
|Visual Feature + kNN||19.33||32.59||53.44||42.93||91.36|
|Visual Feature + SVM||18.79||21.51||35.87||28.82||82.72|
|CNN + Softmax||31.22||31.68||59.52||47.82||98.76|
|CNN + Ranking||26.83||31.93||58.00||46.59||95.06|
|CNN + WARP||35.60||31.65||60.49||48.59||96.29|
|method / metric||per-class recall||per-class precision||overall recall||overall precision|
|Visual Feature + kNN||32.14||22.56||66.98||32.29||95.06|
|Visual Feature + SVM||34.19||18.79||47.15||22.73||96.30|
|CNN + Softmax||48.24||21.98||74.04||35.69||98.76|
|CNN + Ranking||42.48||22.74||72.78||35.08||97.53|
|CNN + WARP||52.03||22.31||75.00||36.16||100.00|
We first report results with respect to the metrics introduced above. In particular, we vary the number of predicted keywords for each image and mainly consider and . Before doing so, however, we must define an upper bound for our evaluation. In the dataset, each image had different numbers of ground-truth tags, which made it hard for us to precisely compute an upper bound for performance with different . For each image, when the number of ground-truth tags was larger than , we randomly chose ground-truth tags and assigned them to that image; when the number of ground-truth tags was smaller than , we assigned all ground-truth tags to that image and randomly chose other tags for that image. We believe this baseline represents the best possible performance when the ground truth is known. The results for assigning 3 keywords per image are reported in Table 1. The results indicate that the deep network achieves a substantial improvement over existing visual-feature-based annotation methods. The CNN+Softmax method outperforms the VisualFeature+SVM baseline by about 10%. Comparing the same CNN network with different loss functions, results show that softmax already gives a very powerful baseline. Although using the pairwise ranking loss does not improve softmax, by using the weighted approximated-ranking loss (WARP) we were able to achieve a substantial improvement over softmax. This is probably because pairwise-ranking is not directly optimizing the top- accuracy, and because WARP pushes classes that are not ranked top heavier, which boosts the performance of rare tags. From these results, we can see that all loss functions achieved comparable overall-recall and overall-precision, but that WARP loss achieved significantly better per-class recall and per-class precision. Results for , which are given in Table 2, show similar trends to .
We also provide a more detailed analysis of per-class recall and per-class precision. The recall for each tags appears in Figure 2, and the precision for each tag in Figure 3. The results for different tags are sorted by the frequency of each tag, in descending order. From these results, we see that the accuracy for frequent tags greater than for infrequent tags. Different losses performed comparably to each other for frequent classes, and WARP worked better than other loss functions for infrequent classes. Finally, we show some image annotation examples in Figure 4. Even though some of the predicted tags for these do not match the ground truth, they are still very meaningful.
5 Discussion and Future Work
In this work, we proposed to use ranking to train deep convolutional neural networks for multilabel image annotation problems. We investigated several different ranking-based loss functions for training the CNN, and found that the weighted approximated-ranking loss works particularly well for multilabel annotation problems. We performed experiments on the largest publicly available multilabel image dataset NUS-WIDE, and demonstrated the effectiveness of using top- ranking to train the network. In the future, we would like to use very large amount of noisy-labeled multilabel images from the Internet (e.g., from Flickr or image searches) to train the network.
-  Kobus Barnard and David Forsyth. Learning the semantics of words and pictures. In ICCV, 2001.
-  Tamara Berg and David Forsyth. Animals on the web. CVPR, 2007.
-  Gustavo Carneiro, Antoni B Chan, Pedro J Moreno, and Nuno Vasconcelos. Supervised learning of semantic classes for image annotation and retrieval. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 29(3):394–410, 2007.
-  Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao. Zheng. Nus-wide: A real-world web image database from national university of singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR’09), Santorini, Greece., July 8-10, 2009.
-  Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc Le, Mark Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Ng. Large scale distributed deep networks. In P. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1232–1240, 2012.
-  Jia Deng, W. Dong, R. Socher, Lijia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
-  Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531, 2013.
Pinar Duygulu, Kobus Barnard, Nando de Freitas, and David Forsyth.
Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary.In ECCV, 2002.
-  R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. Liblinear: A library for large linear classification. JMLR, 2008.
-  Rob Fergus, Antonio Torralba, and Yair Weiss. Semi-supervised learning in gigantic image collections. NIPS, 2009.
-  Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. A multi-view embedding space for internet images, tags, and their semantics. IJCV, 2013.
-  Yunchao Gong and Svetlana Lazebnik. Iterative quantization: An procrustean approach to learning binary codes. CVPR, 2011.
-  Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. ICML, 2013.
-  M. Guillaumin, T. Mensink, J. Verbeek, and C. Schmid. Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation. ICCV, 2009.
-  Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multimodal semi-supervised learning for image classification. CVPR, 2010.
-  Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. Arxiv, 2012.
-  What is the best multi-stage architecture for object recognition? K. jarrett and k. kavukcuoglu and m. a. ranzato and y. lecun. CVPR, 2009.
-  HervJ́égou, M. Douze, Cordelia Schmid, and Patrick Perez. Aggregating local descriptors into a compact image representation. CVPR, 2010.
-  Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, pages 133–142, New York, NY, USA, 2002. ACM.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. NIPS, 2012.
-  Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. CVPR, 2006.
-  Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, and L.D. Jackel. Handwritten digit recognition with a back-propagation network. NIPS, 1990.
-  H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. ICML, 2009.
-  David G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
-  Ameesh Makadia, Vladimir Pavlovic, and Sanjiv Kumar. A new baseline for image annotation. In ECCV, 2008.
-  F. Monay and D. Gatica-Perez. Plsa-based image autoannotation: Constraining the latent space. In ACM Multimedia, 2004.
-  Aude Oliva and Antonio Torralba. Modeling the shape of the scene: a holistic representation of the spatial envelope. IJCV, 2001.
-  Florent Perronnin and Christopher R. Dance. Fisher kernels on visual vocabularies for image categorization. CVPR, 2007.
-  A. Quattoni, M. Collins, and T. Darrell. Learning visual representations using images with captions. CVPR, 2007.
-  N. Rasiwasia, PJ Moreno, and N. Vasconcelos. Bridging the gap: Query by semantic example. IEEE Transactions on Multimedia, 2007.
Koen E. A. van de Sande, Theo Gevers, and Cees G. M. Snoek.
Evaluating color descriptors for object and scene recognition.PAMI, 2010.
-  Li Wan, Matt Zeiler, Sixin Zhang, Yann Lecun, and Rob Fergus. Regularization of neural networks using dropconnect. ICML, 2013.
-  Jinjun Wang, Jianchao Yang, Kai Yu, Fengjun Lv, Thomas Huang, and Yihong Gong. Locality-constrained linear coding for image classification. CVPR, 2010.
-  Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI, 2011.
-  Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. CVPR, 2010.
-  Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching uisng sparse coding for image classification. CVPR, 2009.
-  Matt Zeiler and Rob Fergus. Stochastic pooling for regularization of deep convolutional neural networks. ICLR, 2013.