Heterogeneous Memory Enhanced Multimodal Attention Model for VideoQA
In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.READ FULL TEXT VIEW PDF
How does machine learn to reason about the content of a video in answeri...
We propose a video story question-answering (QA) architecture, Multimoda...
We present MCQA, a learning-based algorithm for multimodal question
The visual explanation of learned representation of models helps to
Visual question answering by using information from multiple modalities ...
Video captioning which automatically translates video clips into natural...
Video Question Answering (Video QA) requires fine-grained understanding ...
Heterogeneous Memory Enhanced Multimodal Attention Model for VideoQA
Video Question Answering (VideoQA) is to learn a model that can infer the correct answer for a given question in human language related to the visual content of a video clip. VideoQA is a challenging computer vision task, as it requires to understand a complex textual question first, and then to figure out the answer that can best associate the semantics to the visual contents in an image sequence.
Recent work [29, 3, 2, 11, 15, 10] proposed to learn models of encoder-decoder structure to tackle the VideoQA problem. A common practice is to use LSTM-based encoders to encode CNN features of video frames and embeddings of question words into encoded visual sequence and word sequence. Proper reasoning is then performed to produce the correct answer, by associating the relevant visual contents with the question. For example, learning soft weights of frames will help attend to events that are queried by the questions, while learning weights of regions in every single frame will help detect details and localize the subjects in the query. The former one aims to find relevant frame-level details by applying temporal attention to encoded image sequence [10, 15, 27]. The latter one aims to find region-level details by spatial attention [26, 12, 29, 2].
Jang et al.  applied spatiotemporal attention mechanism on both spatial and temporal dimension of video features. They also proposed to use both appearance (e.g., VGG ) and motion features (e.g., C3D ) to better represent video frames. Their practice is to make early fusion of the two features and feed the concatenated feature to a video encoder. But such straightforward feature integration leads to suboptimal results. Gao et al.  proposed to replace the early fusion with a more sophisticated co-memory attention mechanism. They used one type of feature to attend to the other and fused the final representations of these two feature types at the final stage. However, this method doesn’t synchronize the attentions detected by appearance and motion features, thus could generate incorrect attentions. Meanwhile, this method will also miss the attention which can be inferred by the combined appearance and motion features, but not individual ones. The principal reason for the existing approaches to fail to identify the correct attention is that they separate feature integration and attention learning steps. To address this challenging problem, we propose a new heterogeneous memory to integrate appearance and motion features and learn spatiotemporal attention simultaneously. In our new memory model, the heterogeneous visual features as multi-input will co-learn the attention to improve the video understanding.
On the other hand, VideoQA becomes very challenging if the question has complex semantics and requires multiple steps of reasoning. Several recent work [32, 15, 5] tried to augment VideoQA with differently embodied memory networks [25, 23, 26]. Xu et al.  proposed to refine the temporal attention over video features word by word with a conventional LSTM question encoder plus an additional LSTM based memory unit to store and update the attention. However, this model is easily trapped into irrelevant local semantics, and cannot understand the question based on the global context. Both Zeng et al.  and Gao et al.  used external memory (memory network  and episodic memory  respectively) to make multiple iterations of inference by interacting the encoded question representation with video features conditioning on current memory contents. However, similar to many other work [26, 10, 2]
, the question representation used in these approaches is only a single feature vector encoded by an LSTM (or GRU) which lacks capability to capture complex semantics in questions such as shown in Fig.1. Thus, it is desired to design a new powerful model for understanding the complex semantics of questions in VideoQA. To tackle this problem, we design novel network architecture to integrate both question encoder and question memory which can augment each other. The question encoder learns meaningful representation of question and the re-designed question memory understands the complex semantics and highlights queried subjects by storing and updating global contexts.
Moreover, we design a multimodal fusion layer which can attend to visual and question hints simultaneously by aligning relevant visual contents with key question words. After gradually refining the joint attention over video and question representations and fusing them with learned soft modality weights, the multi-step reasoning is achieved to infer the correct answer from the complex semantics.
Our major contributions can be summarized as follows: 1) we introduce a heterogeneous external memory module with attentional read and write operations such that the motion and appearance features are integrated to co-learn attention; 2) we utilize the interaction of visual and question features with memory contents to learn global context-aware representations; 3) we design a multimodal fusion layer which can effectively combine visual and question features with softly assigned attentional weights and also support multi-step reasoning; and 4) our proposed model outperforms the state-of-the-art methods on four VideoQA benchmark datasets.
Visual Question Answering (VQA) is an emerging research area [17, 1, 29, 3, 2, 11, 15] to reason the correct answer of a given question which is related to the visual content of an image. Yang et al.  proposed to encode question words into one feature vector which is used as query vector to attend to relevant image regions with stack attention mechanism. Their method supports multi-step reasoning by repeating the query process while refining the query vector. Anderson et al.  proposed to align questions with relevant object proposals in images generated by Faster R-CNN  and compute the visual feature as a weighted average over all proposals. Xiong et al.  proposed to encode image and question features as facts and attend to relevant facts through attention mechanism to generate a contextual vector. Ma et al.  proposed a co-attention model which can attend to not only relevant image regions but also important question words simultaneously. They also suggested to use external memory  to memorize uncommon QA pairs.
Video Question Answering (VideoQA) extends VQA to video domain which aims to infer the correct answer given a relevant question of the visual content of a video clip. VideoQA is considered to be a challenging problem as reasoning on video clip usually requires memorizing contextual information in temporal scale. Many models have been proposed to tackle this problem [31, 10, 32, 27, 30, 5]. Many work [10, 5, 30] utilized both motion (i.e. C3D ) and appearance (i.e. VGG , ResNet ) features to better represent video frames. Similar to the spatial mechanism widely used in VQA methods to find relevant image regions, many VideoQA work [10, 5, 30, 27] applied temporal attention mechanism to attend to most relevant frames of a video clip. Jang  utilized both appearance and motion features as video representations and applied spatial and temporal attention mechanism to attend to both relevant regions of a frame and frames of a video. Xu et al.  proposed to refine the temporal attention over frame features at each question encoding step word by word. Both Zeng et al.  and Gao et al.  proposed to use external memory (Memory Network  and Episodic Memory  respectively) to make multiple iterations of inference by interacting the encoded question feature with video features conditioning on current memory contents. Their memory designs maintain a single hidden state feature of current step and update it through time steps. However, this could hardly establish long-term global context as the hidden state feature is updated at every step. Neither are their models able to synchronize appearance and motion features.
Our model differs from existing work such that 1) we design a heterogeneous external memory module with attentional read and write operations that can efficiently combine motion and appearance features together; 2) we allow interaction of visual and question features with memory contents to construct global context-aware features; and 3) we design a multimodal fusion layer which can effectively combine visual and question features with softly assigned attentional weights and also support multi-step reasoning.
In this section, we illustrate our network architecture for VideoQA. We first introduce the LSTM encoders for video features and question embeddings. Then we elaborate on the design of question memory and heterogeneous video memory. Finally, we demonstrate how our designed multimodal fusion layer can attend to relevant visual and textual hints and combine to form the final answer representation.
Video representation. Following previous work [10, 27, 5], we sample a fixed number of frames (e.g., 35 for TGIF-QA) for all videos in that dataset. We then apply pre-trained ResNet  or VGG  network on video frames to extract video appearance features, and use C3D  network to extract motion features. We denote appearance features as , and motion features as , in which is number of frames. The dimensions of ResNet, VGG and C3D features are 2048, 4096 and 4096. We use two separate LSTM encoders to process motion and appearance features individually first, and late fuse them in the designed memory module which will be discussed in §3.2. In Fig. 2, we highlight the appearance encoder in blue and the motion encoder in orange. The inputs fed into the two encoders are raw CNN motion features and appearance features , and the outputs are encoded motion and appearance features denoted as and .
Question representation. Each VideoQA dataset has a pre-defined vocabulary which is composed of the top most frequent words in the training set. The vocabulary size of each dataset is shown in Table 1. We represent each word as a fixed-length learnable word embedding and initialize with the pre-trained GloVe 300-D  feature. We denote the question embedding as a sequence of word embeddings , in which is number of words in the question. We use another LSTM encoder to process question embedding , as highlighted in red in Fig. 2. The outputs are the encoded text features .
Both motion and appearance visual features are crucial for recognizing the objects and events associated with the questions. Because these two types of features are heterogeneous, the straightforward combination cannot effectively learn the video content. Thus, we propose a new heterogeneous memory to integrate motion and appearance visual features, learn the joint attention, and enhance the spatial-temporal inference.
Different to the standard external memory, our new heterogeneous memory accepts multiple inputs including encoded motion features and appearance features , and uses multiple write heads to determine the content to write. Fig. 3 illustrates the memory structure, which is composed of memory slots and three hidden states , and . We use two hidden states and to determine motion and appearance contents which will be written into memory, and use a separate global hidden state to store and output global context-aware feature which integrates motion and appearance information. We denote the number of memory slots as
, and sigmoid function as. For simplicity, we combine superscript and for identical operations on both motion and appearance features.
Write operation. Firstly we define the motion and appearance content to write to memory at -th time as non-linear mappings from input and previous hidden state
Then we define as the write weights of to each of memory slot given by
satisfying sum to
. Uniquely, we also need to integrate motion and appearance information and make a unified write operation into current memory. Thus we estimate the weightsof motion content , appearance content and current memory content given by
The memory can be updated at each time step by
in which the write weights for memory slots determine how much attention should different slots pay to current inputs, while the modality weights determine which of motion or appearance feature (or none of them if non-informational) from current inputs should the memory pay more attention to. Through this designed memory-write mechanism, we are able to integrate motion and appearance features to learn joint attention, and memorize different spatio-temporal patterns of this video in a synchronized and global context.
Read operation. The next step is to perform an attentional read operation from the memory to update memory hidden states. We define the weights of reading from memory slots as given by
The content read from memory is the weighted sum of each memory slot in which both motion and appearance information has been integrated.
Hidden states update. The final step is to update all three hidden states , and
The global memory hidden state at all time steps will be taken as our final video features. In next section, we will discuss how to generate global question features. In Section 3.4, we will introduce how to interact video and question features for answer inference.
The existing deep learning based VideoQA methods often misunderstand the complex questions because they understand the questions based on local word information. For example, for question “Who drives by a hitchhiking man who is smoking?”, traditional methods are easily trapped by the local words and fail to generate the right attention to the queried person (the driver or the smoker). To address this challenging problem, we introduce the question memory to learn context-aware text knowledge. The question memory can store the sequential text information, learn relevance between words, and understand the question from the global point of view.
We redesign the memory networks [6, 25, 23, 16] to persistently store previous inputs and enable interaction between current inputs and memory contents. As shown in Fig. 4, the memory module is composed of memory slots and memory hidden state . Unlike the heterogeneous memory discussed previously, one hidden state is necessary for the question memory. The inputs to the question memory are the encoded texts .
Write operation. We first define the content to write to the memory at -th time step as which is given by
as a non-linear mapping from current input and previous hidden state to content vector . Then we define the weights of writing to all memory slots such that
satisfying sum to . Then each memory slot is updated by .
Read operation. The next step is to perform attentional read operation from the memory slots . We define the normalized attention weights of reading from memory slots such that
The content read from memory is the weighted sum of each memory slot content .
Hidden state update. The final step of -th iteration is to update the hidden state as
We take the memory hidden state of all time steps as the global context-aware question features which will be used for inference in Section 3.4.
In this section, we design a dedicated multimodal fusion and reasoning module for VideoQA, which can attend to multiple modalities such as visual and textual features, then make multi-step reasoning with refined attention for each modality. Our design is inspired by Hori et al.  which proposed to generate video captions by combining different types of features such as video and audio.
Fig. 5 demonstrates our designed module. The hidden states of video memory and question memory are taken as the input features. The core part is an LSTM controller with its hidden state denoted as . During each iteration of reasoning, the controller attends to different parts of the video features and question features with temporal attention mechanism, and combines the attended features with learned modality weights , and finally updates its own hidden state .
Temporal attention. At -th iteration of reasoning, we first generate two content vectors and by attending to different parts of visual features and question features . The temporal attention weights and are computed by
and shown by the dashed lines in Fig. 5. Then the attended content vectors and the transformed are
Multimodal fusion. The multimodal attention weights are obtained by interacting the previous hidden state with the transformed content vectors
The fused knowledge is computed by the sum of with multimodal attention weights such that
Multi-step reasoning. To complete -th iteration of reasoning, the hidden state of LSTM controller is updated by . This reasoning process is iterated for times and we set . The optimal choice for is discussed in §4.4. The hidden state at last iteration is the final representation of the distilled knowledge. We also apply the standard temporal attention on encoded video features and as in ST-VQA , and concatenate with to form the final answer representation .
We now discuss how to generate the correct answers from answer features .
Multiple-choice task is to choose one correct answer out of candidates. We concatenate the question with each candidate answer individually, and forward each QA pair to obtain the final answer feature , on top of which we use a linear layer to provide scores for all candidate answers in which is the correct answer’s score and the rest are incorrect ones. During training, we minimize the summed pairwise hinge loss  between the positive answer and each negative answer defined as
and train the entire network end-to-end. The intuition of is that the score of the true QA pair should be larger than any negative pair by a margin . During testing, we choose the answer of highest score as the prediction. In Table 1, we list the number of choices for each dataset.
Open-ended task is to choose one correct word as the answer from a pre-defined answer set of size . We apply a linear layer and softmax function upon
to provide probabilities for all candidate answers such thatin which . The training error is measured by cross-entropy loss such that
in which is the ground truth label. By minimizing we can train the entire network end-to-end. In testing phase, the predicted answer is provided by .
We evaluate our model on four benchmark VideoQA datasets and compare with the state-of-the-art techniques.
|Dataset||Feature||Vocab size||Video len||Video num||Question num||Ans size||MC num|
In Table 1, we show the statistics of the four VideoQA benchmark datasets and the experimental settings from their original paper including feature types, vocabulary size, sampled video length, number of videos, size of QA splits, answer set size for open-ended questions, and number of options for multiple-choice questions.
TGIF-QA  contains 165K QA pairs associated with 72K GIF images based on the TGIF dataset . TGIF-QA includes four types of questions: 1) counting the number of occurrences of a given action; 2) recognizing a repeated action given its count; 3) identifying the action happened before or after a given action, and 4) answering image-based questions. MSVD-QA and MSRVTT-QA were proposed by Xu et al.  based on MSVD  and MSVTT  video sets respectively. Five different question types exist in both datasets, including what, who, how, when and where. The questions are open-ended with pre-defined answer sets of size 1000. YouTube2Text-QA  collected three types of questions (what, who and other) from the YouTube2Text  video description corpus. The video source is also MSVD . Both open-ended and multiple-choice tasks exist.
TGIF-QA result. Table 2 summarizes the experiment results of all four tasks (Count,Action,Trans.,FrameQA) on TGIF-QA dataset. We compare with state-of-the-art methods ST-VQA  and Co-Mem  and list the reported accuracy in the original paper. For repetition counting task (column 1), our method achieves the lowest average loss compared with ST-VQA and Co-Mem (4.02 v.s. 4.28 and 4.10). For Action and Trans. tasks (column 2,3), our method significantly outperforms the other two by increasing accuracy from prior best 0.682 and 0.743 to 0.739 and 0.778 respectively. For FrameQA task (column 4), our method also achieves the best accuracy of 0.538 among all three methods, outperforming the Co-Mem by 4.7%.
|Method||Question type and # instances|
MSVD-QA result. Table 3 summarizes the experiment results on MSVD-QA. It’s worth mentioning that there is high class imbalance in both training and test sets, as more than 95% questions are what and who while less than 5% are how, when and where. We list the numbers of their test instances in the table for reference. We compare our model with the ST-VQA , Co-Mem  and current state-of-the-art AMU  on MSVD-QA. We show the reported accuracy of AMU in , while we accommodate the source code of ST-VQA and implement Co-Mem from scratch to obtain their numbers. Our method outperforms all the others on both what and who tasks, and achieves best overall accuracy of 0.337 which is 5.3% better than prior best (0.320). Even though our method slightly underperforms on the How, When and Where questions, the difference are minimal (40,2 and 3) regarding the absolute number of instances due to class imbalance.
MSRVTT-QA result. In Table 4, we compare our model with the ST-VQA , Co-Mem  and AMU  on MSRVTT-QA. Similar to the trend on MSVD-QA, our method outperforms the other models on three major question types (what, who, how), and achieves the best overall accuracy of 0.330.
|Task||Method||Question type and # instances|
YouTube2Text-QA result. In Table 5, we compare our methods with the state-of-the-art r-ANL  on YouTube2Text-QA dataset. It’s worth mentioning that r-ANL utilized frame-level attributes as additional supervision to augment learning while our method does not. For multiple-choice questions, our method significantly outperforms r-ANL on all three types of questions (What, Who, Other) and achieves a better overall accuracy (0.808 v.s. 0.520). For open-ended questions, our method outperforms r-ANL on what queries and slightly underperforms on the other two types. Still, our method achieves a better overall accuracy (0.301 v.s. 0.262). We also report the per-class accuracy to make direct comparison with , and our method is better than r-ANL in this evaluation method.
In Figs. 1 and 6, we demonstrate three QA examples with highlighted key frames and words which are recognized by our designed attention mechanism. For visualization purpose, we extract the visual and textual attention weights from our model (Eq. 12) and plot them with bar charts. Darker color stands for larger weights, showing that the corresponding frame or word is relatively important.
Fig. 1 shows the effectiveness of understanding complex question with our proposed question memory. This question intends to query the female driver though it uses another relative clause to describe the man. Our model focuses on the correct frames in which the female driver is driving in the car and also focuses on the words which describe the woman but not the man. In contrast, ST-VQA  fails to identify the queried person as its simple temporal attention is not able to gather semantic information in the context of a long sentence.
In Fig. 6(a), we provide an example showing that our video memory is learning the most salient frames for the given question while ignoring others. In the first half of the video, it’s difficult to know whether the vegetable is onion or potato, due to the lighting condition and camera view. However, our model smartly pays attention to frames in which the onion is cut into pieces by combining both question words “a man cut” and the motion features, and thus determines the correct object type by onion pieces (but not potato slices) from appearance hint.
Fig. 6(b) shows a typical example illustrating that jointly learning motion and appearance features as our heterogeneous memory design is superior to attending to them separately such as Co-Mem . In this video, a woman is doing yoga in a gym, and there is a barbell rack at the background. Our method successfully associated the woman with the action of exercising, while Co-Mem  incorrectly pays attention to the barbell and fails to utilize motion information as they separately learn motion and appearance attentions.
We perform two ablation studies to investigate the effectiveness of each component of our model. We first study how many iterations of reasoning is sufficient in the designed multimodal fusion layer. After that, we make a comparison of variants of our model to evaluate the contribution of each component.
Reasoning iterations. To understand how many iterations of reasoning are sufficient for our VideoQA tasks, we test different numbers and report their accuracy. The validation accuracy on MSVD-QA dataset increases from 0.298 to 0.306 when the number of reasoning iteration increases from 1 to 3, and seems to saturate at (0.307), while drops to 0.304 at . To balance performance and speed, we choose for our experiments throughout the paper.
Different architectures. To understand the effectiveness of our designed memory module, we compare several variants of our models and evaluate on MSVD-QA and MSRVTT-QA, as shown in Table 6. Early Fusion (EF) is indeed ST-VQA  which concatenates raw video appearance and motion features at an early stage, before feeding into the LSTM encoder. Late Fusion (LF) model uses two separate LSTM encoders to encode video appearance and motion features and then fuses them by concatenation. Episodic Memory (E-M)  is a simplified memory network embodiment and we use it as the visual memory to compare against our design. Visual Memory (V-M) model uses our designed heterogeneous visual memory ( in Fig. 2) to fuse appearance and motion features and generate global context-aware video features. Question Memory (Q-M) model uses our redesigned question memory only ( in Fig. 2) to better capture complex question semantics. Finally, Visual and Question Memory (V+Q M) is our full model which has both visual and question memory.
In Table 6, we observe consistent trend that using memory networks (e.g., E-M,V-M,V+Q) to align and integrate multimodal visual features is generally better than simply concatenating them (e.g., EF,LF). In addition, our designed visual memory (V-M) has shown its strengths over episodic memory (E-M) and other memory types (Table 3-5). Furthermore, using both visual memory and question memory (V+Q) increases the performance by 2-7%.
In this paper, we proposed a novel end-to-end deep learning framework for VideoQA, with designing new external memory modules to better capture global contexts in video frames, complex semantics in questions, and their interactions. A new multimodal fusion layer was designed to fuse visual and textual modalities and perform multi-step reasoning with gradually refined attention. In empirical studies, we visualized the attentions generated by our model to verify its capability of understanding complex questions and attending to salient visual hints. Experimental results on four benchmark VideoQA datasets show that our new approach consistently outperforms state-of-the-art methods.