A typical smartphone user may take hundreds of photos when on vacation or attending an event. Within only a few years, one ends up accumulating dozens of thousands of photos and many hours of video, that capture cherishable moments from one’s past. A recent studyjiang17delving hints that people utilize personal photos as a mean of recovering pieces from their memories. In this paper, we propose MemexQA111Memex was first posited by Bush in 1945 bush1945atlantic as an enlarged intimate supplement to an individual’s memory., a new AI task for question answering: given a collection of photos or videos from a user, the goal is to automatically answer questions that help recover his/her memory about the event captured in the collection. With conversational bots like Siri and Alexa becoming ubiquitous, natural language questions seem to be the best way of going down memory lane. A system utilizing MemexQA can be regarded as a personal assistant of one’s past memories and can answer questions like “When did we last go hiking/camping?” or “ Who had a birthday party in January 2017?”, as long as the answer clues are captured by the user’s photos or videos.
In order to derive an answer for such questions, we rely not only on visual understanding of single photos or videos, but the photo collection as a whole and select the corresponding photos and videos that may support the answers. This extends the scope of classical Visual QA (VQA) antol2015vqa task, which is designed to answer natural language questions for an image, questions that a toddler or an alien would have trouble answering. MemexQA also answers questions based on images. The questions however may possibly require reasoning over multiple images. The answers for MemexQA are not trivial even for an adult to answer manually, as it would require going through a large amount of media first. In our experiments, human participants spend at least 10x longer time to answer a MemexQA question than answering a VQA question. Therefore, we hypothesize that MemexQA models can greatly help users recall their past memories. However, MemexQA is more challenging than classical VQA in the following aspects:
- Collective reasoning
MemexQA adds a localization component across media documents. One needs to inspect multiple items to find clues for the media that support the answer to a given question, while also properly understanding the (visual or other) content of each document.
- Cross-modal reasoning
MemexQA utilizes multimodal knowledge to provide an answer, which may come from the visual content, incomplete titles or tags, timestamps or GPS. It therefore requires some form of reasoning across multimodal information from a large scale dataset. Of course this can be extended to any extra annotations that a collection might have, e.g.
face/person identification, action recognition, sentiment analysis results.
- Dynamic answer space / memory
The answer space for MemexQA is somewhere between the closed set of most current VQA tasks and open-ended QA, as it is conditioned on the database. However, not only this space is not predefined and different among users, it also grows as the database increases. This is far larger than any current VQA task. Furthermore, MemexQA has an information retrieval component and takes into account the ever-growing dynamic memory of a user’s media collection. This implicitly connects the task to personalization as the same question would give totally different answers for 2 different users.
In this paper we present a novel approach towards solving the MemexQA task. Given the absence of data to tackle this problem, we first crowdsourced questions and answers for personal Flickr images huang2016visual , organized in user albums. That results in theMemexQA dataset (https://memexqa.cs.cmu.edu/) consisting of 13,591 personal photos from 630 albums and 101 Flickr users and 20,860 crowdsourced questions. Examples of questions, answers and photos from the dataset are illustrated in Fig. 1.
We further propose MemexNet, an united network architecture for image, text and video question answering. The proposed MemexNet is an end-to-end trainable network which is learned together with off-the-shelf image/video engines and is of configurable modular networks for answer inference. To summarize, our contributions are the following: 1) We introduce a new AI task, MemexQA, as well as a large, croudsourced dataset that consists of question and answer pairs corresponding about not just a single image but also a collection of images. 2) We propose a novel approach to tackle MemexQA via a unified, end-to-end trainable model. Extensive experimental results on the MemexQA dataset and demonstrate that the proposed method outperforms all strong baselines and yields the state-of-the-art on this very challenging dataset. The promising results on TextQA and VideoQA suggest the efficacy and scalability of the proposed method across various QA tasks.
We consider MemexQA to be a multi-disciplinary AI task as it combines computer vision, natural language processing and information retrieval and extraction. We believe that this combination will enable us to scale to real-world applications and ever-growing user media collections. Our study also provides a first but promising solution to answer questions based on large-scale video repository.
2 Related Work
Visual QA VQA was first proposed by Antol et al. in antol2015vqa and has received a large amount of attention in recent years yang2016stacked ; xiong2016dynamic ; LuXPS16 ; LuYBP16 ; FukuiPYRDR16 ; xu2015show ; wu2016ask ; WangWSHD16 . The goal of VQA is to answer open-ended questions about a given image. The challenge of VQA lies in understanding both the image semantics xu2015show and the knowledge base wu2016ask ; WangWSHD16 . The difference between MemexQA and VQA roots in the scalability, where MemexQA models must digest the whole collection of personal images and videos before answer the questions.
Text QA An important contributor to the recent advancement in Text QA has been the availability of various datasets. Our MemexQA is motivated by Watson ferrucci2010building and Memory Networks weston2014memory , of which both learn a text based matching model for either open domain or close domain questions. Our MemexQA considers problem in a different field with multi-modalities including images, videos, and meta data. However, it can be applied to Text QA problems too. The recently released SQuAD rajpurkar2016squad dataset with over 100,000 question-answer pairs has sparkled many studies on end-to-end machine comprehension neural models. Match-LSTM wang2016machine proposed a method to match the context with the question and then utilized a modified Pointer Network to find the answer positions. DCN xiong2016dynamicb introduced a dynamic co-attention network to capture the interactions between the context and the question and used a dynamic decoder to predict the beginning and the ending points of the answer. MPCM wang2016multi proposed a multi-perspective context matching model. The bi-directional attention flow (BiDAF) seo2016bidirectional used a bi-directional attention mechanism to get question-aware context representation, while a variant of Memory Network dhingra2016gated ; shen2016reasonet
was used to calculate the attention vector between query and context. We compare some of these recent models in the Text QA task in Section5.2
Memex The term “Memex” was first posited by V. Bush in 1945 as an enlarged intimate supplement to an individual’s memory. Bush envisioned the Memex as a device in which individuals would compress and store all of their information, mechanized so that it may be consulted with exceeding speed and flexibility bush1945atlantic . The concept of the Memex influenced the development of early hypertext systems, eventually leading to the creation of the World Wide Web davies2011still and the personal search system gemmell2002mylifebits . The proposed MemexQA advances the traditional Memex system to answer questions about the personal photo and video content.
3 The MemexQA dataset
This section introduces the MemexQA dataset and compares it to existing VQA datasets. The MemexQA dataset consists of 13,591 personal photos from 101 Flickr users. Theses personal photos capture a variety significant movements of their lives such as marriage proposals, wedding ceremonies, birthday/farewell parties, family reunions, commencements, etc. We collect comprehensive multimodal information about each photo, which includes a timestamp, GPS (if any), a photo title, photo tags, album information as well as photo captions from the SIND data huang2016visual . Through AMT (Amazon Mechanical Turk), we collect a total of 20,860 questions and 417,200 multiple choice answers. The annotators are instructed to first watch and understand all photos of a single Flickr user, treat these photos as if they were their own, and write meaningful questions&answers that they might ask to recall their memories about the event in these photos. The annotator provides a correct answer as well as one or more evidential photos to justify the answer. See Fig. 2. In total 33,297 evidential photos are collected, i.e. 1.6 photos per question. The collected data are of decent quality, where the inter-human agreement is 91%, which is comparable to 83.3% on the VQA dataset antol2015vqa . Motivated by a recent study jiang17delving , we focus on five types of questions: “what”, “who”, “where”, “when” and “how many”. As the study showed, the query terms in the first 4W category account for more than 60% of personal search traffic. We choose not to include the “show me” questions, as such questions can be addressed by the classical image/video retrieval method.
It is worth noting that MemexQA is significantly more expensive to collect than VQA. In VQA, annotators write a question by looking at only a single photo. In MemexQA, to write a question, they need to not only inspect tens of photos, but also consult the supporting multimodal information. As a result, we estimate that it takes 10 times longer for annotators to write (9610 seconds) and answer (623 seconds) a MemexQA question than a VQA question. Given those restrictions, we consider the MemexQA dataset to be sufficiently large as a first testbed on which different methods for this novel task can be compared. The experimental results in Section 5 substantiate this claim.
3.1 Data Collection
Generally, the MemexQA data are collected on AMT in 3 steps: QA collection, candidate answers generation, and QA cleanup. We acknowledge the following assumption for collecting the data: we assume all questions should be answerable by any individual and not just the owner of the photos. Therefore, information that is not captured by the photo or metadata should not be used in answering questions. This may lead to lower recall yet a more objective approach for evaluation.
Questions & Answers. We select the Flickr users that have at least 4 albums from the SIND dataset huang2016visual . This results in a collection of 13,951 Flickr photos from 630 albums of 101 real Flickr users. The QA data collection tasks are conducted on AMT, an online crowd-sourcing platform. The online AMT annotators are instructed to write 1) important questions 2) objective questions, and 3) concise answers about the main topic in each album. Each annotator is only asked to write questions about a single Flickr user. Specifically, they write 5 questions/answers about the photos in a single photo album, and for each answer they provide one or more evidential photos to justify their answers. This process is first repeated for all albums separately. Then the photos from all albums are shown, and they write questions about photos across multiple albums, and similarly provide evidential photos for each answer. The frequent questions about a single album are: “where did we go”, “what did we do”, “what was the wealth like”, etc. The frequent questions about multiple are “what was the last time”, “how many times did”.
Candidate Answers. Following zhu2016visual7w , MemexQA employs both human workers and an automatic method to generate a pool of candidate answers. For the “what” question, a pool of candidate answers is automatically generated based on the answer of similar questions in the MemexQA and VQA datasets. For other types of questions, a pool of candidate answers is obtained by randomly selecting relevant user metadata. All candidate answers are then inspected by annotators to ensure there is only one correct answer for each question.
QA Cleanup To reduce the ambiguous and low-quality QA, each photo album is independently annotated by at least 5 AMT workers. The QA along with candidate answers are verified by 3 workers, where they are asked to select the correct answer from the 4 choices. The workers report unreasonable QAs, when they find subjective questions/answers, or answers with 1+ correct choices. The collected data are of decent quality. The sampled inter-human agreement, which measures the percentage of the questions having same answer cross different AMT workers, is 0.9. This number is comparable or even better than that on existing VQA datasets antol2015vqa ; zhu2016visual7w .
3.2 Comparison to VQA datasets
MemexQA is a real-world QA dataset that requires multimodal reasoning based on both vision and language. It compliments existing VQA datasets by allowing users to ask multimodal questions over a collection of photos. Table 1 compares MemexQA to representative VQA datasets antol2015vqa ; zhu2016visual7w . As we see, the MemexQA dataset encompasses the most comprehensive information including evidential photos, Time GPS, and metadata. More importantly, it is the only effort that has questions grounded on multiple images. It further has the longest question and answers, which, according to zhu2016visual7w , is a good indicator for high-quality data.
MemexQA is a realistic dataset. It comprises personal photos of 101 real users and 20k faithful questions that users asked to recall their memories. We found that the statistics about MemexQA are closer to the statistics about Flickr photos, from which VQA and Visual7W datasets are crawled. For example, FlickrOR in Table 1 calculates the overlap ratio between the frequent search terms in the Flickr search logs jiang17delving and the frequent answers in QA datasets. As we see, MemexQA’s ratio is 6% higher than others, which suggests its answers are closer to user interests. Similarly, the question category distribution of MemexQA also seems more realistic (see supplementary materials).
MemexQA is a multimodal dataset. The comprehensive multimodal information results in a ideal testbed for experimenting the vision+language QA problem. Fig. 2 illustrates the diverse sources from which an answer can be derived, e.g. from title+time (Q1), vision+time (Q2, Q3, Q6), vision+OCR+GPS (Q4), vision+time+album info (Q5). As we show in Section 5, a solution to MemexQA may also be applicable to generic QA problems including text, image or video QA.
This section introduces MemexNet, a novel network architecture inspired by the Watson QA system ferrucci2010building . MemexNet is designed for MemexQA, but may also be applied to text or video based question answering. As illustrated in Fig. 3, there are three major modules in a MemexNet: the question understanding, search engine, and answer inference module.
Question understanding. The query understanding module is responsible for encoding the question for subsequent search and inference tasks. Specifically, it contains two subnetworks, a question encoder and a query encoder . Given a question , the question encoder embeds it into a latent state vector , which expresses the question content and the estimated category to which its answer belongs, such as a date, number, name, place or action, scene, etc.
will be passed to the answer inference module by a residual connection (shown in Fig.3). The query encoder discovers underlying concepts in the given question and produces a query embedding as the input to the content search engine. Usually, is a sparse vector over a set of relevant concepts in the vocabulary. For the multimedia cases, the query encoder maps a natural language sentence to a set of relevant concepts that exist in the pre-trained concept vocabulary jiang17delving . For pure textual retrieval, e.g. for TextQA, the query encoder can be replaced by a simple lemmatization model, since out-of-vocabulary words are rare. For MemexNet we employ LSTM networks as our question encoder, SkipGram models mikolov2013distributed and VQE (Visual Query Embedding) networks jiang17delving as the query encoder. The supplementary materials discuss implementation details.
Search engine. The second module is a content-based search engine that can automatically index and search the content of images/videos. Given a collection of samples (images or videos), it outputs a list of samples ranked by , where is the embedded query and represents the retrieval model, which retrieves images/videos based on the embedded query . Instead of learning everything from scratch, we propose to leverage the off-the-shelf retrieval models for training the MemexNet. This is mainly to reduce the learning space for answer inference. The state-of-the-art visual search engines sivic2003video ; jiang2015bridging allow us not only to search from large scale image or video corpus efficiently, but also to perform complex queries which include the time, geolocation, or subtle relation between concepts. In this work, we choose not to fine-tune the retrieval model but, instead, to learn attention weights over the retrieved samples. Indeed, an alternative approach is to learn a -nearest neighbor function from scratch, as suggested in memory networks chandar2016hierarchical . However, we found that it might not improve the accuracy, as it might be too difficult to learn a neighborhood function that maps from text to images/videos in the current datasets.
Answer inference. The answer inference module is a modular network andreas2016neural , in which various networks can be plugged in/out for different QA tasks. For example, the Bi-Directional Attention Flow (BiDAF) layers are used for Text QA in Section 5.2. To tackle the answer inference problem in MemexQA, we develop a novel MMLookupNet (MultiModal Lookup Network), shown in Fig. 4. Following VQA, in this paper, we regard MemexQA as a classification problem. For each question , we have the correct answer coded in the one-hot vector representation of
answer classes. We propose to estimate the probability of an answer class using the top-retrieved samples b y:
where , are the top- samples ranked by ; represents the outputs of MMLookupNet, and is the CNN-based image feature for the sample . in Eq. (1) represents the classification function, which, in MemexNet is implemented as two fully-connected layers.
For training, as shown in Eq. (1), the lookup embedding , the question embedding and the image CNN features are all fed to the classification layers to minimize the softmax cross-entropy loss, in which only the losses to the class in the multiple choices are considered.
The rest of the section will discuss the lookup embedding . The retrieved samples come with various metadata such as the timestamp, title, concepts, etc., as shown in Fig. 4. Let the superscript index the -th modality of . For example, is “05/16/2017” in Fig. 4). MMLookupNet represents the multimodal information of the retrieved samples by concatenating two embeddings: for modalities and for matched classes. Specifically, same modalities share the same indexed by the modality index . is calculated from:
where is the answer text from the -th class and is a kernel function measuring the similarity between the text of and . For this paper, we simply compute as the proportion of exactly matched non-stop stemmed words. We introduce a pilot answer named “$” where . The pilot answer can gate non-match answers, i.e. the cases where the best matched answer still has a very low score.
Let index the top- retrieved samples and index the modality. Denote for the embedding about the -th modality of the th sample, where is the vector concatenation operator. Intuitively, samples at different rank positions should have different importance. To incorporate this information, at the answer inference step, we introduce attentions over the retrieved samples. As shown in Fig. 4, are sequentially fed into the LSTM network, where the attention weight is calculated from:
where is the LSTM output at the ranked position , and are parameters to learn. The context vector is used to represent the LSTM outputs with attention over rank positions. Similarly, we will use to index the context vector for different modalities. Finally, MMLookupNet represents the modality by a concatenated vector of , and , where . Then the above vectors of all modalities are concatenated as to represent the top- sample. See Fig. 4.
introduces an embedding space for models to learn to either lookup or infer an answer based on available information. When multiple matched answers are found, the model is expected to learn salient weights to attend to samples at appropriate ranked positions by , and meanwhile to differentiate modalities by the modality embedding . For example, the embedding for the “time” modality is expected to receive higher weights for “when” questions. When no matched answers are found, the model still can infer an answer based on other information such as the question embedding, image CNN features and the modality embedding.
In this section, we empirically verify MemexNet on three tasks: MemexQA, TextQA and VideoQA.
5.1 MemexNet on MemexQA
Human Experiments We first examine the human performance on MemexQA. We are interested in measuring 1) how well human can perform in the MemexQA task, 2) what is the contribution of each modality in helping users answer questions, and 3) how long does it take for humans to answer a MemexQA question. We conduct a set of controlled experiments, where AMT workers are asked to select an answer from 4 choices given different information, which includes Questions, Answers, Images with timestamps and gps (if any), and Metadata (titles and descriptions). Table 2 reports the results. As we see, humans manage to correctly guess 50% of the correct answers using common sense. With all information, the accuracy reaches 0.93, which is comparable to 0.83 on VQA antol2015vqa and 0.97 on Visual7W zhu2016visual7w . The accuracy without images drops significantly, suggesting MemexQA requires substantial visual understanding and reasoning. Besides, experiments show that on average it takes 10 times longer (62s) for humans to answer a MemexQA question than a VQA question (5.5s zhu2016visual7w ). This suggests that the automatic MemexQA model, which can be consulted with exceeding speed and flexibility, can help provide enlarged supplement to one’s memory.
Model Experiments We then study the performance of state-of-the-art models on the MemexQA task222The experiments were performed on the MemexQA dataset v0.8 version.. By default, all models use photos, timestamps, gps, and titles. Models are learned on a training set of 14,156 randomly selected QA pairs, and tested on a set of 3,539 QAs. The remaining QAs are used for validation. The most frequent 7,236 answers are selected as class labels, which cover 99% of the multiple choices in the dataset. To represent an image, we use the output of the last layer of ResNet he2016deep , PCA-reduced to 300 dimensions (dimensionality reduction speeds up convergence).
We compare MemexNet with both classical and state-of-the-art VQA models: Logistic Regression predicts the answer from a concatenation of image, question and metadata features. Question and metadata features are represented by the averaged word embedding from the pre-trained SkipGram model mikolov2013distributed . Embedding
is similar to Logistic Regression except it learns the word embedding for questions and metedata.BoW derives the answer by a concatenation of the image feature and the bag-of-words representation of the question and metadata. LSTM represents the LSTM VQA model in malinowski2015ask , where the concatenation of question embedding, average image features, and average metadata embeddings are fed into every LSTM unit. LSTM + Att represents the previous LSTM model with attention mechanisms. LSTM w./ Multi-Channels is the RNN network described in antol2015vqa , where the sequence of question words, metadata, and photos are sequentially fed into three LSTM networks, respectively. The element-wise dot product of all hidden states are used for classification. For MemexNet, Eq. (1) is used, where top- is set to 2. We extract concepts by the Google Vision API (https://cloud.google.com/vision/), employ E-Lamp Lite jiang2015bridging as the content search engine. and the SkipGram model as the query encoder. See the details in supplementary materials.
Table 2 reports the overall performance. As we see, MemexNet outperforms all baseline methods, with statistically significant differences. The result indicates MemexNet is a promising network for vision and language reasoning on this new task. The significant gap between the human and model accuracy, however, indicates that MemexQA is still a very challenging AI task. To analyze the contribution of the MMLookupNet, we replace it with the average embedding of concepts and metadata, and report the accuracy in Table 2. As we see, the notable performance drop, especially on “what” and “when” questions, suggests MMLookupNet is beneficial in training MemexNet.
|LSTM + Att||0.700||0.356||0.461||0.391||0.451||0.433|
|LSTM w/. Multi-Channels||0.717||0.282||0.347||0.300||0.342||0.356|
|MemexNet w/o. MMLookupNet||0.677||0.378||0.382||0.353||0.430||0.418|
5.2 MemexNet on TextQA
This subsection evaluates MemexNet on TextQA on the SQuAD dataset rajpurkar2016squad . We employ a lemmatization model as our query encoder and the bi-directional LSTM networks (word embedding + char CNN) as our question encoder. In this data, we treat each sentence as a photo, each paragraph as an album, and retrieve the top 5 sentences relevant for each question using the language retrieval model with the JM smoother zhai2004study . The last 3 layers in the BiDAF model seo2016bidirectional are used as our answer inference networks. We compare MemexNet with recent TextQA networks in terms of score in Table 3. As we see, although MemexNet is not designed for this task and only uses 5 sentences, it achieves accuracy which is comparable or better than the recent text QA models.
5.3 MemexNet on VideoQA
This subsection verifies the efficacy and scalability on the VideoQA task. To this end, we apply MemexNet on YFCC100M, one of the largest public personal image and video collections thomee2016yfcc100m , and ask questions about 800K videos in the collection. We choose not to use the user-generated metadata (titles or descriptions) to access the QA performance based only on video content understanding. We build our VideoQA model using similar configurations as in Section 5.1. For such a large scale problem, an important metric is the runtime speed, which reflects the scalability and efficiency in real applications. To measure the run timespeed, we use a single core of Intel Xeon 2.53GHz CPU with 32GB memory. On average, it takes about 1.3 seconds to answer a question over 800 thousand videos purely based on the video content, suggesting that the proposed MemexNet is scalable for real-world large-scale QA problems. See some example questions in our demo video in the supplementary materials. Since the annotated ground truth is impossible to obtain for all the videos, we ask two human experts to ask our systems 25 questions and evaluate the results. The estimated averaged accuracy is 0.52. The breakdown of results is included in the supplementary material.
6 Conclusions and Future Work
In this paper, we propose MemexQA, a multi-disciplinary AI task and present a large dataset of real personal photos and crowd-sourced questions/answers to tackle that problem. We further present MemexNet, a unified, end-to-end trainable network architecture for image, text and video question answering. The experimental results on the MemexQA dataset demonstrate MemexNet’s state-of-the-art performance on this challenging task. The promising results on TextQA and VideoQA suggest MemexNet is applicable to QA tasks across various domains and efficient for large scale image and video albums. We consider our work as a first step towards solving this new interdisciplinary AI problem. Our future works include personalized QA, fine-grained photo and album understanding.
This work was partially supported by Yahoo InMind project and Flickr Computer Vision and Machine Learning group.
-  J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In CVPR, 2016.
-  S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. Vqa: Visual question answering. In ICCV, 2015.
-  V. Bush. The atlantic monthly. As we may think, 176(1):101–108, 1945.
-  S. Chandar, S. Ahn, H. Larochelle, P. Vincent, G. Tesauro, and Y. Bengio. Hierarchical memory networks. arXiv preprint arXiv:1605.07427, 2016.
-  S. Davies. Still building the memex. Communications of the ACM, 54(2):80–88, 2011.
-  B. Dhingra, H. Liu, W. W. Cohen, and R. Salakhutdinov. Gated-attention readers for text comprehension. arXiv preprint arXiv:1606.01549, 2016.
-  D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, et al. Building watson: An overview of the deepqa project. AI magazine, 31(3):59–79, 2010.
-  A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. CoRR, abs/1606.01847, 2016.
-  J. Gemmell, G. Bell, R. Lueder, S. Drucker, and C. Wong. Mylifebits: fulfilling the memex vision. In MM, 2002.
-  K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
-  T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, J. Devlin, A. Agrawal, R. Girshick, X. He, P. Kohli, D. Batra, et al. Visual storytelling. In NAACL, 2016.
-  L. Jiang, L. Cao, Y. Kalantidis, S. Farfade, J. Tang, and A. G. Hauptmann. Delving deep into personal photo and video search. In WSDM, 2017.
-  L. Jiang, S.-I. Yu, D. Meng, T. Mitamura, and A. G. Hauptmann. Bridging the ultimate semantic gap: A semantic search engine for internet videos. In ICMR, 2015.
-  J. Lu, C. Xiong, D. Parikh, and R. Socher. Knowing when to look: Adaptive attention via A visual sentinel for image captioning. CoRR, abs/1612.01887, 2016.
-  J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. NIPS, 2016.
M. Malinowski, M. Rohrbach, and M. Fritz.
Ask your neurons: A neural-based approach to answering questions about images.In ICCV, 2015.
-  T. Mikolov and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
-  P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
-  M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603, 2016.
-  Y. Shen, P.-S. Huang, J. Gao, and W. Chen. Reasonet: Learning to stop reading in machine comprehension. arXiv preprint arXiv:1609.05284, 2016.
-  J. Sivic, A. Zisserman, et al. Video google: A text retrieval approach to object matching in videos. In ICCV, 2003.
-  B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.
-  P. Wang, Q. Wu, C. Shen, A. van den Hengel, and A. R. Dick. FVQA: fact-based visual question answering. CoRR, abs/1606.05433, 2016.
-  S. Wang and J. Jiang. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905, 2016.
-  Z. Wang, H. Mi, W. Hamza, and R. Florian. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211, 2016.
-  J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014.
-  Q. Wu, P. Wang, C. Shen, A. Dick, and A. van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In CVPR, 2016.
-  C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. arXiv, 1603, 2016.
-  C. Xiong, V. Zhong, and R. Socher. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604, 2016.
-  K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
-  Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In CVPR, 2016.
-  C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to information retrieval. TOIS, 22(2):179–214, 2004.
-  Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. 0swering in images. In CVPR, 2016.