Automatic description of visual content has lately received a lot of interest in our community. Multiple works have successfully addressed the image captioning problem [6, 16, 17, 35]. Many of the proposed methods rely on Long-Short Term Memory networks (LSTMs) . In the meanwhile, two large-scale movie description datasets have been proposed, namely MPII Movie Description (MPII-MD)  and Montreal Video Annotation Dataset (M-VAD) . Both are based on movies with associated textual descriptions and allow studying the problem how to generate movie description for visually disabled people. Works addressing these datasets [28, 33, 39] show that they are indeed challenging in terms of visual recognition and automatic description. This results in a significantly lower performance then on simpler video datasets (e.g. MSVD ), but a detailed analysis of the difficulties is missing. In this work we address this by taking a closer look at the performance of existing methods on the movie description task.
This work contributes a) an approach to build robust visual classifiers which distinguish verbs, objects, and places extracted from weak sentence annotations; b) based on the visual classifiers we evaluate different design choices to train an LSTM for generating descriptions. This outperforms related work on the MPII-MD dataset, both using automatic and human evaluation; c) we perform a detailed analysis of prior work and our approach to understand the challenges of the movie description task.
2 Related Work
. Many of them rely on Recurrent Neural Networks (RNNs) and in particular on Long-Short Term Memory networks (LSTMs). Also new datasets have been released, Flickr30k and MS COCO Captions , where  additionally presents a standardized setup for image captioning evaluation. There are also attempts to analyze the performance of recent methods. E.g.  compares them with respect to the novelty of generated descriptions and additionally proposes a nearest neighbor baseline that improves over recent methods.
In the past video description has been addressed in semi-realistic settings [1, 18], on a small scale [4, 11, 30] or in constrained scenarios like cooking [27, 29]. Most works (with a few exceptions, e.g. ) study the task of describing a short clip with a single sentence.  first proposed to describe videos using an LSTM, relying on precomputed CRF scores from . 
extended this work to extract CNN features from frames which are max-pooled over time. They show the benefit of pre-training the LSTM network for image captioning and fine-tuning it to video description. proposes a framework that consists of a 2-D and/or 3-D CNN and the LSTM is trained jointly with a visual-semantic embedding to ensure better coherence between video and text.  jointly addresses the language generation and video/language retrieval tasks by learning a joint embedding model for a deep video model and compositional semantic language model.
Recently two large-scale movie description datasets have been proposed, MPII Movie Description (MPII-MD)  and Montreal Video Annotation Dataset (M-VAD) . Given that they are based on movies, they cover a much broader domain then previous video description datasets. Consequently they are much more varied and challenging with respect to the visual content and the associated description. They also do not have any additional annotations, as e.g. TACoS Multi-Level , thus one has to rely on the weak annotations of the sentence descriptions. To handle this challenging scenario  proposes an attention based model which selects the most relevant temporal segments in a video and incorporates 3-D CNN and generates a sentence using an LSTM.  proposes an encoder-decoder framework, where a single LSTM encodes the input video frame by frame and decodes it into a sentence, outperforming . Our approach for sentence generation is most similar to 
and we also rely on their LSTM implementation based on Caffe. However, we analyze different aspects and variants of this architecture for movie description. To extract labels from sentences we rely on the semantic parser of , however we treat the labels differently to handle the weak supervision (see Section 3.1). We show that this improves over  and .
In this section we present our two-step approach to video description. The first step performs visual recognition, while the second step generates textual descriptions. For the visual recognition we propose to use the visual classifiers trained according to the labels’ semantics and “visuality”. For the language generation we rely on a LSTM network which has been successfully used for image and video description [6, 33]. We discuss various design choices for building and training the LSTM. An overview of our approach is given in Figure 1.
3.1 Visual Labels for Robust Visual Classifiers
For training we rely on a parallel corpus of videos and weak sentence annotations. As in  we parse the sentences to obtain a set of labels (single words or short phrases, e.g. look up) to train our visual classifiers. However, in contrast to  we do not want to keep all of these initial labels as they are noisy, but select only visual ones which actually can be robustly recognized.
Avoiding parser failure.
Not all sentences can be parsed successfully, as e.g. some sentences are incomplete or grammatically incorrect. To avoid loosing the potential labels in these sentences, we match our set of initial labels to the sentences which the parser failed to process.
Our labels correspond to different semantic groups. In this work we consider three most important groups: verbs (actions), objects and places, as they are typically visual. One could also consider e.g. groups like mood or emotions, which are naturally harder for visual recognition. We propose to treat each label group independently. First, we rely on a different representation for the each semantic groups, which is targeted to the specific group. Namely we use the activity recognition feature Improved Dense Trajectories (DT)  for verbs, large scale object detector responses (LSDA) 
for objects and scene classification scores (PLACES) for places. Second, we train one-vs-all SVM classifiers for each group separately. The intuition behind this is to discard “wrong negatives” (e.g. using object “bed” as negative for place “bedroom”).
Now, how do we select visual labels for our semantic groups? In order to find the verbs among the labels we rely on the semantic parser of . Next, we look up the list of “places” used in  and search for corresponding words among our labels. We look up the object classes used in  and search for these “objects”, as well as their base forms (e.g. “domestic cat” and “cat”). We discard all the labels that do not belong to any of our three groups of interest as we assume that they are likely not visual and thus are difficult to recognize. Finally, we discard labels which the classifiers could not learn, as these are likely to be noisy or not visual. For this we require the classifiers to have have minimum area under the ROC-curve (Receiver Operating Characteristic).
3.2 LSTM for Sentence Generation
(a), at each time step, an LSTM generates a word and receives the visual classifiers (input-vis) as well as as the previous generated word (input-lang) as input. To handle natural words we encode each word with a one-hot-vector according to their index in a dictionary and a lower dimensional embedding. The embedding is jointly learned during training of the LSTM. compares three variants: (a) an encoder-decoder architecture, (b) a decoder architecture with visual max predictions, and (c) a decoder architecture with visual probabilistic predictions. In this work we rely on variant (c) which was shown to work best as it can rely on the richest visual input. We analyze the following aspects for this architecture:
We compare a 1-layer architecture with a 2-layer architecture. In the 2-layer architecture, the output of the first layer is used as input for the second layer (Figure 2b) and was used by  for video description. Additionally we also compare to a 2-layer factored architecture , where the first layer only gets the language as input and the second gets the output of the first layer as well as the visual input.
To learn a more robust network which is less likely to overfit we rely on a dropout . Using dropout a ratio of randomly selected units is set to 0 during training (while all others are multiplied with ). We explore different ways to place dropout in the network, i.e. either for language input (lang-drop) or visual (vis-drop) input only, for both inputs (concat-drop) or for the LSTM output (lstm-drop), see Figure 2(d). While the default dropout ratio is , we evaluate the effect of different ratios.
By default we rely on a step-based learning strategy, where a learning rate is halved after a certain number of steps. We find the best learning rate and step size on the validation set. Additionally we compare this to a polynomial learning strategy, where the learning rate is continuously decreased. The polynomial learning strategy has been shown to give good results faster without tweaking step size for GoogleNet implemented by Sergio Guadarrama in Caffe .
In this section we first analyze our approach on the MPII-MD  dataset and explore different design choices. Then, we compare our best system to prior work.
4.1 Analysis of our approach
We build on the labels discovered by our semantic parser  and additionally match these labels to sentences which the parser failed to process. To be able to learn classifiers we select the labels that appear at least 30 times, resulting in 1,263 labels. The parser additionally tells us whether the label is a verb. We use the visual features (DT, LSDA, PLACES) provided with the MPII-MD dataset . The LSTM output/hidden unit as well as memory cell have each 500 dimensions. We train the SVM classifiers on the Training set (56,861 clips). We evaluate our method on the validation set (4,930 clips) using the METEOR  score, which, according to [7, 32], supersedes other popular measures, such as BLEU , ROUGE , in terms of agreement with human judgments. The authors of CIDEr  showed that METEOR also outperforms CIDEr when the number of references is small and in the case of MPII-MD we have typically only a single reference.
|Baseline: all labels treated the same way|
|(5) Verbs(DT), Others(LSDA)||1328||7.08||7.27|
|(6) Verbs(DT), Places(PLACES), Others(LSDA)||1328||7.09||7.39|
|(7) Verbs(DT), Places(PLACES), Objects(LSDA)||913||7.10||7.48|
|(8) + restriction to labels with||263||7.41||7.54|
|Baseline: all labels treated the same way, labels from (8)|
4.1.1 Robust visual classifiers.
In a first set of experiments we analyze our proposal to consider groups of labels to learn different classifiers and also to use different visual representations for these groups (see Section 3.1). Table 1 we evaluate our generated sentences using different input features to the LSTM. In our baseline, in the top part of Table 1, we treat all labels equally, i.e. we use the same visual descriptors for all labels. The PLACES feature is best with 7.1 METEOR. Combination by stacking all features (DT + LSDA + PLACES) improves further to 7.24 METEOR.
The second part of the table demonstrates the effect of introducing different semantic label groups. We first split the labels into “Verbs” and all remaining. Given that some labels appear in both roles, the total number of labels increases to 1328. We analyze two settings of training the classifiers. In the case of “Retrieved” we retrieve the classifier scores from the general classifiers trained in the previous step. “Trained” corresponds to training the SVMs specifically for each label type (e.g. for “verbs”). Next, we further divide the non-verbal labels into “Places” and “Others”, and finally into “Places” and “Objects”. We discard the unused labels and end up with 913 labels. Out of these labels, we select the labels where the classifier obtains a ROC higher or equal to 0.7 (threshold selected on the validation set). After this we obtain 263 labels and the best performance in the “Trained” setting. To support our intuition about the importance of the label discrimination (i.e. using different features for different semantic groups of labels), we propose another baseline (last line in the table). Here we use the same set of 263 labels but provide the same feature for all of them, namely the best performing combination DT + LSDA + PLACES. As we see, this results in an inferior performance.
We make several observations from Table 1 which lead to robust visual classifiers from the weak sentence annotations. a) It is beneficial to select features based on the label semantics. b) Training one-vs-all SVMs for specific label groups consistently improves the performance as it avoids “wrong” negatives. c) Focusing on more “visual” labels helps: we reduce the LSTM input dimensionality to 263 while improving the performance.
4.1.2 LSTM architectures.
We start with examining the architecture, where we explore different configurations of LSTM and dropout layers. Table 1(a) shows the performance of three different networks: “1 layer”, “2 layers unfactored” and “2 layers factored” introduced in Section 3.2. As we see, the “1 layer” and “2 layers unfactored” perform equally well, while “2 layers factored” is inferior to them. In following experiments we use the simplest “1 layer” network. We then compare different dropout placements as illustrated in (Figure 1(b)). We obtain the best result when applying dropout after the LSTM layer (“lstm-drop”), while having no dropout or applying it only to language leads to stronger over-fitting to the visual features. Putting dropout after the LSTM (and prior to a final prediction layer) makes the entire system more robust. As for the best dropout ratio, we find that 0.5 works best with lstm-dropout Table 1(c).
We compare different learning rates and learning strategies in Tables 2(a) and 2(b). We find that the best learning rate in the step-based learning is 0.01, while step 4000 slightly improves over step 2000 (which we used in Table 1). We explore an alternative learning strategy, namely decreasing learning rate according to a polynomial decay. We experiment with different exponents (0.5 and 0.7) and numbers of iterations (25K and 10K), using the base-learning rate 0.01. Our results show that the step-based learning is superior to the polynomial learning.
|1 net: lr 0.01, step 2000, iter=25,000||7.54|
|ensemble of 3 nets||7.52|
|1 net: lr 0.01, step 4000, iter=25,000||7.59|
|ensemble of 3 nets||7.68|
|1 net: lr 0.01, step 4000, iter=15,000||7.55|
|ensemble of 3 nets||7.72|
In most of experiments we trained our networks for 25,000 iterations. After looking at the METEOR performance for intermediate iterations we found that for the step size 4000 at iteration 15,000 we achieve best performance overall. Additionally we train multiple LSTMs with different random orderings of the training data. In our experiments we combine three in an ensemble, averaging the resulting word predictions. In most cases the ensemble improves over the single networks in terms of METEOR score (see Table 4).
To summarize, the most important aspects that decrease over-fitting and lead to a better sentence generation are: (a) a correct learning rate and step size, (b) dropout after the LSTM layer, (c) choosing the training iteration based on METEOR score as opposed to only looking at the LSTM accuracy/loss which can be misleading, and (d) building ensembles of multiple networks with different random initializations. In the following section we evaluate our best ensemble (last line of Table 4) on the test set of MPII-MD.
4.2 Comparison to related work
We compare the best method of , the recently proposed method S2VT  and our proposed “Visual Labels”-LSTM on the test set of the MPII-MD dataset (6,578 clips). We report all popular automatic evaluation measures, CIDEr , BLEU , ROUGE  and METEOR , computed using the evaluation code of . We also perform a human evaluation, by randomly selecting 1300 video snippets and asking AMT turkers to rank three systems (the best SMT of , S2VT  and ours) with respect to Correctness, Grammar and Relevance, similar to .
|Automatic Score||Human evaluation: rank|
|Best SMT of ||8.14||0.47||13.21||5.59||2.11||2.39||2.08|
Table 5 summarizes the results on the test set of MPII-MD. While we rely on identical features and similar labels as , we significantly improve the performance in all automatic measures, specifically by 1.44 METEOR points. Moreover, we improve over the recent approach of , which also uses LSTM to generate video descriptions. Exploring different strategies to label selection and classifier training, as well as various LSTM configurations allows to obtain best result to date on the MPII-MD dataset. Human evaluation mainly agrees with the automatic measures. We outperform both prior works in terms of Correctness and Relevance, however we lose to S2VT in terms of Grammar. This is due to the fact that S2VT produces overall shorter (7.4 versus 8.7 words per sentence) and simpler sentences, while our system generates longer sentences and therefore has higher chances to make mistakes.
We also propose a retrieval upperbound (last line in Table 5). For every test sentence we retrieve the closest training sentence according to the METEOR. The rather low METEOR score of 19.43 reflects the difficulty of the dataset.
A closer look at the sentences produced by all three methods gives us additional insights. An interesting characteristic is the output vocabulary size, which is 94 for , 86 for  and 605 for our method, while the test set contains 6422 unique words. This clearly shows a higher diversity of our output. Among the words generated by our system and absent in the outputs of others are such verbs as grab, drive, sip, climb, follow, objects as suit, chair, cigarette, mirror, bottle and places as kitchen, corridor, restaurant. We showcase some qualitative results in Figure 3. Here, e.g. the verb pour, object drink and place courtyard only appear in our output. We attribute this, on one hand, to our diverse and robust visual classifiers. On the other hand, the architecture and parameter choices of our LSTM allow us to learn better correspondance between words and visual classifiers’ scores.
|SMT ||Someone is a man, someone is a man.|
|S2VT ||Someone looks at him, someone turns to someone.|
|Our||Someone is standing in the crowd,|
|a little man with a little smile.|
|Reference||Someone, back in elf guise, is trying to calm the kids.|
|SMT ||The car is a water of the water.|
|S2VT ||On the door, opens the door opens.|
|Our||The fellowship are in the courtyard.|
|Reference||They cross the quadrangle below and run along the cloister.|
|SMT ||Someone is down the door,|
|someone is a back of the door, and someone is a door.|
|S2VT ||Someone shakes his head and looks at someone.|
|Our||Someone takes a drink and pours it into the water.|
|Reference||Someone grabs a vodka bottle standing open on the counter|
|and liberally pours some on the hand.|
Despite the recent advances in the video description domain, including our proposed approach, the video description performance on the movie description datasets (MPII-MD  and M-VAD ) remains relatively low. In this section we want to take a closer look at three methods, best SMT of , S2VT  and ours, in order to understand where these methods succeed and where they fail. In the following we evaluate all three methods on the MPII-MD test set.
5.1 Difficulty versus performance
As the first study we suggest to sort the reference sentences (from the test set) by difficulty, where difficulty is defined in multiple ways.
Sentence length and Word frequency. Two of the simplest sentence difficulty measures are its length and average frequency of words. When sorting the data by difficulty (increasing sentence length or decreasing average word frequency), we find that all three methods have the same tendency to obtain lower METEOR score as the difficulty increases (Figures 3(a) and 3(b)). For the word frequency the correlation is stronger. Our method consistently outperforms the other two, most notable as the difficulty increases.
METEOR score per sentence. (a) Test set sorted by Textual NN score (decreasing). (b) Test set sorted by Visual kNN score,(decreasing). Shown values are smoothed with a mean filter of size 500.
Textual and Visual Nearest Neighbors. Next, for each reference test sentence we search for the closest training sentence (in terms of the METEOR score). We use the obtained best scores to sort the reference sentences by textual difficulty, i.e. the “easy” sentences are more likely to be retrieved. If we consider all training sentences, we obtain a Textual Nearest Neighbor. We sort the test sentences according to these scores (decreasing) and plot the performance of three methods in Figure 4(a). All methods “agree” and ours is best throughout the difficulty range, in particular in the more challenging part of the plot. We can also use visual features to find the Nearest Neighbors in the Training set, select the best one (in terms of the METEOR score) and use this score to sort the reference sentences. We call this a Visual k Nearest Neighbor. The intuition behind it is to consider a video clip as visually “easy” if the most similar training clips also have similar descriptions (the “difficult” clip might have no close visual neighbours). We rely on our best visual representation (8) from Table 1 and similarity measure to define the Visual kNN and sort the reference sentences according to it with (Figure 4(b)). We see a clear correlation between the visual difficulty and the performance of all methods (Figure 4(b)).
Summary. a) All methods perform better on shorter, common sentences and our method notably wins on longer sentences. b) Our method also wins on sentences that are more difficult to retrieve. c) Visual difficulty, defined by similarity and representation (8) from Table 1, strongly correlates with the performance of all methods. (d) When comparing all four plots (Figures 3(a) and 3(b), Figures 4(a) and 4(b)), we find that the strongest correlation between the methods’ performance and the difficulty is observed for the Textual difficulty, while the least correlation we observe for the Sentence length.
5.2 Semantic analysis
|creation||5.69||hit||make||do||walk through||come up|
WordNet Verb Topics. We closer analyze the test sentences with respect to different verbs. For this we rely on WordNet topics (high level entries in the WordNet ontology, e.g. “motion”, “perception”, “competition”, “emotion”), defined for most synsets in WordNet . We obtain the sense information from the semantic parser of , thus senses might be noisy. We showcase the 5 most frequent verbs for each topic in Table 6. We select sentences with a single verb, group them according to the verb topic and compute an average METEOR score for each topic, see Figure 6. We find that our method is best for all topics except “communication”, where  wins. The most frequent verbs in this topic are “look up” and “nod”, which are also frequent in the dataset and in the sentences produced by . The best performing topic, “cognition”, is highly biased to “look at” verb. The most frequent topics, “motion” and “contact”, which are also visual (e.g. “turn”, “walk”, “open”, “sit”), are nevertheless quite challenging, which we attribute to their high diversity (see their entropy w.r.t. different verbs and their frequencies in Table 6). At the same time “perception” is far less diverse and mainly focuses on verbs like “look” or “stare”, which are quite frequent in the dataset, resulting in better performance. Topics with more abstract verbs (e.g. “be”, “have”, “start”) tend to get lower scores.
Top 100 best and worst sentences. We look at 100 Test sentences, where our method obtains highest and lowest METEOR scores. Out of 100 best sentences 44 contain the verb “look” (including verb phrases such as “look at”). The other frequent verbs are “walk”, “turn”, “smile”, “nod”, “shake”, “stare”, “sit”, i.e. mainly visual verbs. Overall the sentences are simple and common. Among the 100 lowest scoring sentences we observe more diversity: 12 sentences contain no verb, 10 mention unusual words (specific to the movie), 24 contain no subject, 29 have a non-human subject. Altogether this leads to a lower performance, in particular, as most training sentences contain “Someone” as subject and generated sentences are biased towards it.
Summary. a) The test sentences that mention the verb “look” (and similar) get higher METEOR scores due to their high frequency in the dataset. b) The sentences with more “visual” verbs tend to get higher scores. c) The sentences without verbs (e.g. describing a scene), without subjects or with non-human subjects get lower scores, which can be explained by a dataset bias towards “Someone” as subject.
We propose an approach to automatic movie description which trains visual classifiers and uses the classifier scores as input to LSTM. To handle the weak sentence annotations we rely on three main ingredients. First, we distinguish three semantic groups of labels (verbs, objects and places), second we train them discriminatively, removing potentially noisy negatives, and third, we select only a small number of the most reliable classifiers. For sentence generation we show the benefits of exploring different LSTM architectures and learning configurations. As the result we obtain the highest performance on the MPII-MD dataset as shown by all automatic evaluation measures and extensive human evaluation.
We analyze the challenges in the movie description task using our and two prior works. We find that the factors which contribute to higher performance include: presence of frequent words, sentence length and simplicity as well as presence of “visual” verbs (e.g. “nod”, “walk”, “sit”, “smile”). Textual and visual difficulties of sentences/clips strongly correlate with the performance of all methods. We observe a high bias in the data towards humans as subjects and verbs similar to “look”. Future work has to focus on dealing with less frequent words and handle less visual descriptions. This potentially requires to consider external text corpora, modalities other than video, such as audio and dialog, and to look across multiple sentences. This would allow exploiting long- and short-range context and thus understanding and describing the story of the movie.
Marcus Rohrbach was supported by a fellowship within the FITweltweit-Program of the German Academic Exchange Service (DAAD). The authors thank Niket Tandon for help with the WordNet Topics analysis.
Barbu, A., Bridge, A., Burchill, Z., Coroian, D., Dickinson, S., Fidler, S., Michaux, A., Mussman, S., Narayanaswamy, S., Salvi, D., Schmidt, L., Shangguan, J., Siskind, J.M., Waggoner, J., Wang, S., Wei, J., Yin, Y., Zhang, Z.: Video in sentences out. In: Proceedings of the conference on Uncertainty in Artificial Intelligence (UAI) (2012)
-  Chen, D., Dolan, W.: Collecting highly parallel data for paraphrase evaluation. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (2011)
-  Chen, X., Fang, H., Lin, T., Vedantam, R., Gupta, S., Doll r, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server. arXiv:1504.00325 (2015)
Das, P., Xu, C., Doell, R., Corso, J.: Thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)
-  Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., Mitchell, M.: Language models for image captioning: The quirks and what works. arXiv:1505.01809 (2015)
-  Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Elliott, D., Keller, F.: Image description using visual dependency representations. In: EMNLP. pp. 1292–1302 (2013)
-  Fang, H., Gupta, S., Iandola, F.N., Srivastava, R., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J.C., Zitnick, C.L., Zweig, G.: From captions to visual concepts and back. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Farhadi, A., Hejrati, M., Sadeghi, M., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: Every picture tells a story: Generating sentences from images. In: Proceedings of the European Conference on Computer Vision (ECCV) (2010)
-  Fellbaum, C.: WordNet: An Electronical Lexical Database. The MIT Press (1998)
-  Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., Saenko, K.: Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shoot recognition. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2013)
-  Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580 (2012)
-  Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation (1997)
-  Hoffman, J., Guadarrama, S., Tzeng, E., Donahue, J., Girshick, R., Darrell, T., Saenko, K.: LSDA: Large scale detection through adaptation. In: Advances in Neural Information Processing Systems (NIPS) (2014)
-  Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093 (2014)
-  Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics (TACL) (2015)
-  Kojima, A., Tamura, T., Fukunaga, K.: Natural language description of human activities from video images based on concept hierarchy of actions. International Journal of Computer Vision (IJCV) (2002)
-  Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: Baby talk: Understanding and generating simple image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)
-  Kuznetsova, P., Ordonez, V., Berg, T.L., Hill, U.C., Choi, Y.: Treetalk: Composition and compression of trees for image descriptions. In: Transactions of the Association for Computational Linguistics (TACL) (2014)
-  Lavie, M.D.A.: Meteor universal: Language specific translation evaluation for any target language. ACL 2014 p. 376 (2014)
Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. pp. 74–81 (2004)
-  Mao, J., Xu, W., Yang, Y., Wang, J., Yuille, A.L.: Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv:1412.6632 (2014)
-  Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A.C., Berg, T.L., III, H.D.: Midge: Generating image descriptions from computer vision detections. In: Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL) (2012)
-  Pan, Y., Mei, T., Yao, T., Li, H., Rui, Y.: Jointly modeling embedding and translation to bridge video and language. arXiv:1505.01861 (2015)
-  Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (2002)
-  Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Pinkal, M., Schiele, B.: Coherent multi-sentence video description with variable level of detail. In: Proceedings of the German Confeence on Pattern Recognition (GCPR) (2014)
-  Rohrbach, A., Rohrbach, M., Tandon, N., Schiele, B.: A dataset for movie description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., Schiele, B.: Translating video content to natural language descriptions. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2013)
-  Thomason, J., Venugopalan, S., Guadarrama, S., Saenko, K., Mooney, R.J.: Integrating language and vision to generate natural language descriptions of videos in the wild. In: Proceedings of the International Conference on Computational Linguistics (COLING) (2014)
-  Torabi, A., Pal, C., Larochelle, H., Courville, A.: Using descriptive video services to create a large data source for video annotation research. arXiv:1503.01070v1 (2015)
-  Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., Saenko, K.: Sequence to sequence – video to text. arXiv:1505.00487 (2015)
-  Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., Saenko, K.: Translating videos to natural language using deep recurrent neural networks. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) (2015)
-  Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Wang, H., Schmid, C.: Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2013)
-  Xu, K., Ba, J., Kiros, R., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. arXiv:1502.03044 (2015)
-  Xu, R., Xiong, C., Chen, W., Corso, J.J.: Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In: Proceedings of the Conference on Artificial Intelligence (AAAI) (2015)
-  Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., Courville, A.: Describing videos by exploiting temporal structure. arXiv:1502.08029v4 (2015)
-  Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL) 2, 67–78 (2014)
Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning Deep Features for Scene Recognition using Places Database. Advances in Neural Information Processing Systems (NIPS) (2014)