Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.READ FULL TEXT VIEW PDF
Representation learning for text and images has been extensively studied in recent years. Recurrent neural networks (RNNs) are often used to represent sentences or phrases[Sutskever et al.2014, Kiros et al.2015]
, and convolutional neural networks (CNNs) have shown to work best to represent images[Donahue et al.2013, He et al.2015]. For tasks such as visual question answering (VQA) and visual grounding, most approaches require joining the representation of both modalities. For combining the two vector representations (multimodal pooling), current approaches in VQA or grounding rely on concatenating vectors or applying element-wise sum or product. While this generates a joint representation, it might not be expressive enough to fully capture the complex associations between the two different modalities.
In this paper, we propose to rely on Multimodal Compact Bilinear pooling (MCB) to get a joint representation. Bilinear pooling computes the outer product between two vectors, which allows, in contrast to element-wise product, a multiplicative interaction between all elements of both vectors. Bilinear pooling models [Tenenbaum and Freeman2000] have recently been shown to be beneficial for fine-grained classification for vision only tasks [Lin et al.2015]. However, given their high dimensionality (), bilinear pooling has so far not been widely used. In this paper, we adopt the idea from gao16cvpr which shows how to efficiently compress bilinear pooling for a single modality. In this work, we discuss and extensively evaluate the extension to the multimodal case for text and visual modalities. As shown in Figure 1, Multimodal Compact Bilinear pooling (MCB) is approximated by randomly projecting the image and text representations to a higher dimensional space (using Count Sketch [Charikar et al.2002]
) and then convolving both vectors efficiently by using element-wise product in Fast Fourier Transform (FFT) space. We use MCB to predict answers for the VQA task and locations for the visual grounding task. For open-ended question answering, we present an architecture for VQA which uses MCB twice, once to predict spatial attention and the second time to predict the answer. For multiple-choice question answering we introduce a third MCB to relate the encoded answer to the question-image space. Additionally, we discuss the benefit of attention maps and additional training data for the VQA task. To summarize, MCB is evaluated on two tasks, four datasets, and with a diverse set of ablations and comparisons to the state-of-the-art.
Current approaches to multimodal pooling involve element-wise operations or vector concatenation. In the visual question answering domain, a number of models have been proposed. Simpler models such as iBOWIMG baseline [Zhou et al.2015] use concatenation and fully connected layers to combine the image and question modalities. Stacked Attention Networks [Yang et al.2015] and Spatial Memory Networks [Xu et al.2015] use LSTMs or extract soft-attention on the image features, but ultimately use element-wise product or element-wise sum to merge modalities. D-NMN [Andreas et al.2016a] introduced REINFORCE to dynamically create a network and use element-wise product to join attentions and element-wise sum predict answers. Dynamic Memory Networks (DMN) [Xiong et al.2016] pool the image and question with element-wise product and sum, attending to part of the image and question with an Episodic Memory Module [Kumar et al.2016]. DPPnet [Noh et al.2015] creates a Parameter Prediction Network which learns to predict the parameters of the second to last visual recognition layer dynamically from the question. Similar to this work, DPPnet allows multiplicative interactions between the visual and question encodings. lu2016hiecoatt recently proposed a model that extracts multiple co-attentions on the image and question and combines the co-attentions in a hierarchical manner using element-wise sum, concatenation, and fully connected layers.
For the visual grounding task, rohrbach16arxiv propose an approach where the language phrase embedding is concatenated with the visual features in order to predict the attention weights over multiple bounding box proposals. Similarly, hu16arxiv concatenate phrase embeddings with visual features at different spatial locations to obtain a segmentation.
Bilinear pooling has been applied to the fine-grained visual recognition task. lin15iccv use two CNNs to extract features from an image and combine the resulting vectors using an outer product, which is fully connected to an output layer. gao16cvpr address the space and time complexity of bilinear features by viewing the bilinear transformation as a polynomial kernel. pham2013tensorsketch describe a method to approximate the polynomial kernel using Count Sketches and convolutions.
In order to model similarities between two modalities, many prior works have learned joint multimodal spaces, or embeddings. Some of such embeddings are based on Canonical Correlation Analysis [Hardoon et al.2004] e.g. [Gong et al.2014, Klein et al.2015, Plummer et al.2015], linear models with ranking loss [Frome et al.2013, Karpathy and Fei-Fei2015, Socher et al.2014, Weston et al.2011]
or non-linear deep learning models[Kiros et al.2014, Mao et al.2015, Ngiam et al.2011]. Our multimodal compact bilinear pooling can be seen as a complementary operation that allows us to capture different interactions between two modalities more expressively than e.g. concatenation. Consequently, many embedding learning approaches could benefit from incorporating such interactions.
For the task of visual question answering (VQA) or visual grounding, we have to predict the most likely answer or location for a given image x and question or phrase q. This can be formulated as
with parameters and the set of answers or locations . For an image embedding (i.e. a CNN) and question embedding (i.e. an LSTM), we are interested in getting a good joint representation by pooling both representations. With a multimodal pooling that encodes the relationship between and
well, it becomes easier to learn a classifier for Equation (1).
In this section, we first discuss our multimodal pooling for combining representations from different modalities into a single representation (Sec. 3.1) and then detail our architectures for VQA (Sec. 3.2) and visual grounding (Sec. 3.3), further explaining how we predict with the given image representation and text representation .
Bilinear models [Tenenbaum and Freeman2000] take the outer product of two vectors and and learn a model (here linear), i.e. , where denotes the outer product () and denotes linearizing the matrix in a vector. As discussed in the introduction, bilinear pooling is interesting because it allows all elements of both vectors to interact with each other in a multiplicative way. However, the high dimensional representation (i.e. when and are large) leads to an infeasible number of parameters to learn in . For example, we use and for VQA. thus would have 12.5 billion parameters, which leads to very high memory consumption and high computation times.
We thus need a method that projects the outer product to a lower dimensional space and also avoids computing the outer product directly. As suggested by gao16cvpr for a single modality, we rely on the Count Sketch projection function [Charikar et al.2002], which projects a vector to . We initialize two vectors and : contains either or for each index, and maps each index in the input to an index in the output . Both and
are initialized randomly from a uniform distribution and remain constant for future invocations of count sketch.is initialized as a zero vector. For every element its destination index is looked up using , and is added to . See lines 1-9 and 12-16 in Algorithm 1.
This allows us to project the outer product to a lower dimensional space, which reduces the number of parameters in . To avoid computing the outer product explicitly, pham2013tensorsketch showed that the count sketch of the outer product of two vectors can be expressed as convolution of both count sketches: , where is the convolution operator. Additionally, the convolution theorem states that convolution in the time domain is equivalent to element-wise product in the frequency domain. The convolution can be rewritten as , where refers to element-wise product. These ideas are summarized in Figure 2 and formalized in Algorithm 1
, which is based on the Tensor Sketch algorithm of pham2013tensorsketch. We invoke the algorithm withand . We note that this easily extends and remains efficient for more than two multi-modal inputs as the combination happens as element-wise product.
In VQA, the input to the model is an image and a question, and the goal is to answer the question. Our model extracts representations for the image and the question, pools the vectors using MCB, and arrives at the answer by treating the problem as a multi-class classification problem with 3,000 possible classes.
We extract image features using a 152-layer Residual Network [He et al.2015]
that is pretrained on ImageNet data[Deng et al.2009]. Images are resized to , and we use the output of the layer (“pool5”) before the 1000-way classifier. We then perform normalization on the 2048-D vector.
Input questions are first tokenized into words, and the words are one-hot encoded and passed through a learned embedding layer. The tanh nonlinearity is used after the embedding. The embedding layer is followed by a 2-layer LSTM with 1024 units in each layer. The outputs of each LSTM layer are concatenated to form a 2048-D vector.
The two vectors are then passed through MCB. The MCB is followed by an element-wise signed square-root and normalization. After MCB pooling, a fully connected layer connects the resulting 16,000-D multimodal representation to the 3,000 top answers.
To incorporate spatial information, we use soft attention on our MCB pooling method. Explored by [Xu et al.2015] for image captioning and by [Xu and Saenko2016] and [Yang et al.2015] for VQA, the soft attention mechanism can be easily integrated in our model.
For each spatial grid location in the visual representation (i.e. last convolutional layer of ResNet [res5c], last convolutional layer of VGG [conv5]), we use MCB pooling to merge the slice of the visual feature with the language representation. As depicted in Figure 3, after the pooling we use two convolutional layers to predict the attention weight for each grid location. We apply softmax to produce a normalized soft attention map. We then take a weighted sum of the spatial vectors using the attention map to create the attended visual representation. We also experiment with generating multiple attention maps to allow the model to make multiple “glimpses” which are concatenated before being merged with the language representation through another MCB pooling for prediction. Predicting attention maps with MCB pooling allows the model to effectively learn how to attend to salient locations based on both the visual and language representations.
For VQA with multiple choices, we can additionally embed the answers. We base our approach on the proposed MCB with attention. As can be seen from Figure 4, to deal with multiple variable-length answer choices, each choice is encoded using a word embedding and LSTM layers whose weights are shared across the candidates. In addition to using MCB with attention, we use an additional MCB pooling to merge the encoded answer choices with the multimodal representation of the original pipeline. The resulting embedding is projected to a classification vector with a dimension equal to the number of answers.
We base our grounding approach on the fully-supervised version of GroundeR [Rohrbach et al.2016]. The overview of our model is shown in Figure 5. The input to the model is a query natural language phrase and an image along with multiple proposal bounding boxes. The goal is to predict a bounding box which corresponds to the query phrase. We replace the concatenation of the visual representation and the encoded phrase in GroundeR with MCB to combine both modalities. In contrast to rohrbach16arxiv, we include a linear embedding of the visual representation and
normalization of both input modalities, instead of batch normalization[Ioffe and Szegedy2015], which we found to be beneficial when using MCB for the grounding task.
We evaluate the benefit of MCB with a diverse set of ablations on two visual question answering datasets.
The Visual Question Answering (VQA) real-image dataset [Antol et al.2015] consists of approximately 200,000 MSCOCO images [Lin et al.2014], with 3 questions per image and 10 answers per question. There are 3 data splits: train (80K images), validation (40K images), and test (80K images). Additionally, there is a 25% subset of test named test-dev. Accuracies for ablation experiments in this paper are reported on the test-dev data split. We use the VQA tool provided by antol2015vqa for evaluation. We conducted most of our experiments on the open-ended real-image task. In Table 4, we also report our multiple-choice real-image scores.
The Visual Genome dataset [Krishna et al.2016] uses 108,249 images from the intersection of YFCC100M [Thomee et al.2015] and MSCOCO. For each image, an average of 17 question-answer pairs are collected. There are 1.7 million QA pairs of the 6W question types (what, where, when, who, why, and how). Compared to the VQA dataset, Visual Genome represents a more balanced distribution of the 6W question types. Moreover, the average question and answer lengths for Visual Genome are larger than the VQA dataset. To leverage the Visual Genome dataset as additional training data, we remove all the unnecessary words such as ”a”, ”the”, and ”it is” from the answers to decrease the length of the answers and extract QA pairs whose answers are single-worded. The extracted data is filtered again based on the answer vocabulary space created from the VQA dataset, leaving us with additional 1M image-QA triplets.
The Visual7W dataset [Zhu et al.2016] is a part of the Visual Genome. Visual7W adds a 7th which question category to accommodate visual answers, but we only evaluate the models on the Telling task which involves 6W questions. The natural language answers in Visual7W are in a multiple-choice format and each question comes with four answer candidates, with only one being the correct answer. Visual7W is composed of 47,300 images from MSCOCO and there are a total of 139,868 QA pairs.
|Concatenation + FC||58.40|
|Concatenation + FC + FC||57.10|
|Element-wise Product + FC||56.44|
|Element-wise Product + FC + FC||57.88|
|Full Bilinear (K)||58.46|
|Element-wise Product with VGG-19||55.97|
|MCB (K) with VGG-19||57.05|
|Concatenation + FC with Attention||58.36|
|MCB (K) with Attention||62.50|
We use the Adam solver with , , . We use dropout after the LSTM layers and in fully connected layers. For the experiments in Table 1 and 2, we train on the VQA train split, validate on the VQA validation split, and report results on the VQA test-dev split. We use early stopping: if the validation score does not improve for 50,000 iterations, we stop training and evaluate the best iteration on test-dev.
For the Visual7W task, we use the same hyperparameters and training settings as in the VQA experiments. We use the splits from[Zhu et al.2016] to train, validate, and test our models. We also compute accuracies on this data using their evaluation code.
For VQA multiple choice, we train the open-ended models and take the argmax over the multiple choice answers at test time. For Visual7W, we use the answer encoding as described in Sec. 3.2.
|Zhu et al.||51.5||57.0||75.0||59.5||55.5||49.8||54.3|
|Open Ended||MC||Open Ended||MC|
|MCB + Genome||81.7||36.6||51.5||62.3||66.4||-||-||-||-||-|
|MCB + Att.||82.2||37.7||54.8||64.2||68.6||-||-||-||-||-|
|MCB + Att. + GloVe||82.5||37.6||55.6||64.7||69.1||-||-||-||-||-|
|MCB + Att. + Genome||81.7||38.2||57.0||65.1||69.5||-||-||-||-||-|
|MCB + Att. + GloVe + Genome||82.3||37.2||57.4||65.4||69.9||-||-||-||-||-|
|Ensemble of 7 Att. models||83.4||39.8||58.5||66.7||70.2||83.2||39.5||58.0||66.5||70.1|
|Naver Labs (challenge 2nd)||83.5||39.8||54.8||64.9||69.4||83.3||38.7||54.6||64.8||69.3|
|HieCoAtt [Lu et al.2016]||79.7||38.7||51.7||61.8||65.8||-||-||-||62.1||66.1|
|DMN+ [Xiong et al.2016]||80.5||36.8||48.3||60.3||-||-||-||-||60.4||-|
|FDA [Ilievski et al.2016]||81.1||36.2||45.8||59.2||-||-||-||-||59.5||-|
|D-NMN [Andreas et al.2016a]||81.1||38.6||45.5||59.4||-||-||-||-||59.4||-|
|AMA [Wu et al.2016]||81.0||38.4||45.2||59.2||-||81.1||37.1||45.8||59.4||-|
|SAN [Yang et al.2015]||79.3||36.6||46.1||58.7||-||-||-||-||58.9||-|
|NMN [Andreas et al.2016b]||81.2||38.0||44.0||58.6||-||81.2||37.7||44.0||58.7||-|
|AYN [Malinowski et al.2016]||78.4||36.4||46.3||58.4||-||78.2||36.3||46.3||58.4||-|
|SMem [Xu and Saenko2016]||80.9||37.3||43.1||58.0||-||80.9||37.5||43.5||58.2||-|
|VQA team [Antol et al.2015]||80.5||36.8||43.1||57.8||62.7||80.6||36.5||43.7||58.2||63.1|
|DPPnet [Noh et al.2015]||80.7||37.2||41.7||57.2||-||80.3||36.9||42.2||57.4||-|
|iBOWIMG [Zhou et al.2015]||76.5||35.0||42.6||55.7||-||76.8||35.0||42.6||55.9||62.0|
We compare the performance of non-bilinear and bilinear pooling methods in Table 1. We see that MCB pooling outperforms all non-bilinear pooling methods, such as eltwise sum, concatenation, and eltwise product.
One could argue that the compact bilinear method simply has more parameters than the non-bilinear pooling methods, which contributes to its performance. We compensated for this by stacking fully connected layers (with 4096 units per layer, ReLU activation, and dropout) after the non-bilinear pooling methods to increase their number of parameters. However, even with similar parameter budgets, non-bilinear methods could not achieve the same accuracy as the MCB method. For example, the “Concatenation + FC + FC” pooling method has approximatelymillion parameters, which matches the 48 million parameters available in MCB with . However, the performance of the “Concatenation + FC + FC” method is only 57.10% compared to MCB’s 59.83%.
Section 2 in Table 1 also shows that compact bilinear pooling has no impact on accuracy compared to full bilinear pooling. Section 3 in Table 1 demonstrates that the MCB brings improvements regardless of the image CNN used. We primarily use ResNet-152 in this paper, but MCB also improves performance if VGG-19 is used. Section 4 in Table 1
shows that our soft attention model works best with MCB pooling. In fact, attending to the Concatenation + FC layer has the same performance as not using attention at all, while attending to the MCB layer improves performance by 2.67 points.
Table 2 compares different values of , the output dimensionality of the multimodal compact bilinear feature. Approximating the bilinear feature with a 16,000-D vector yields the highest accuracy.
We also evaluated models with multiple attention maps or channels. One attenion map achieves 64.67%, two 65.08% and four 64.24% accuracy (trained on train+val). Visual inspection of the generated attention maps reveals that an ensembling or smoothing effect occurs when using multiple maps.
Table 3 presents results for the Visual7W multiple-choice QA task. The MCB with attention model outperforms the previous state-of-the-art by 7.9 points overall and performs better in almost every category.
Table 4 compares our approach with the state-of-the-art on VQA test set. Our best single model uses MCB pooling with two attention maps. Additionally, we augment our training data with images and QA pairs from the Visual Genome dataset. We also concatenate the learned word embedding with pretrained GloVe vectors [Pennington et al.2014].
Each model in our ensemble of 7 models uses MCB with attention. Some of the models were trained with data from Visual Genome, and some were trained with two attention maps. This ensemble is 1.8 points above the next best approach on the VQA open-ended task and 0.8 points above the next best approach on the multiple-choice task (on Test-dev). Even without ensembles, our “MCB + Genome + Att. + GloVe” model still outperforms the next best result by 0.5 points, with an accuracy of 65.4% versus 64.9% on the open-ended task (on Test-dev).
|plummer16arxiv111plummer16arxiv achieve higher accuracy of 50.89% when taking into account box size and color. We believe our approach would also benefit from such additional features.||43.84|
|Element-wise Product + Conv||47.86|
|Element-wise Product + Conv||27.98|
|What vegetable is the dog chewing on?
|What kind of dog is this?
|What kind of flooring does the room have?
|What color is the traffic light?
|Is this an urban area?
|Where are the buildings?
MCB: in background
GT: on left
|MCB||Eltwise Product + Conv||MCB||Eltwise Product + Conv|
|A tattooed woman with a green dress and yellow backpack holding a water bottle is walking across the street.||A dog distracts his owner from working at her computer.|
We evaluate our visual grounding approach on two datasets. The first is Flickr30k Entities [Plummer et al.2015] which consists of 31K images from Flickr30k dataset [Hodosh et al.2014] with 244K phrases localized with bounding boxes. We follow the experimental setup of rohrbach16arxiv, e.g. we use the same Selective Search [Uijlings et al.2013] object proposals and the Fast R-CNN [Girshick2015] fine-tuned VGG16 features [Simonyan and Zisserman2014]. The second dataset is ReferItGame [Kazemzadeh et al.2014], which contains 20K images from IAPR TC-12 dataset [Grubinger et al.2006] with segmented regions from SAIAPR-12 dataset [Escalante et al.2010] and 120K associated natural language referring expressions. For ReferItGame we follow the experimental setup of hu16cvpr and rely on their ground-truth bounding boxes extracted around the segmentation masks. We use the Edge Box [Zitnick and Dollár2014] object proposals and visual features (VGG16 combined with the spatial features, which encode bounding box relative position) from hu16cvpr.
In all experiments we use Adam solver [Kingma and Ba2014] with learning rate . The embedding size is both for visual and language embeddings. We use in the MCB pooling, which we found to work best for the visual grounding task. The accuracy is measured as percentage of query phrases which have been localized correctly. The phrase is localized correctly if the predicted bounding box overlaps with the ground-truth bounding box by more than intersection over union (IOU).
Tables 5 and 6 summarize our results in the visual grounding task. We present multiple ablations of our proposed architecture. First, we replace the MCB with simple concatenation of the embedded visual feature and the embedded phrase, resulting in on the Flickr30k Entities and on the ReferItGame datasets. The results can be improved by replacing the concatenation with the element-wise product of both embedded features ( and ). We can further slightly increase the performance by introducing additional 2048-D convolution after the element-wise product ( and ). However, even with fewer parameters, our MCB pooling significantly improves over this baseline on both datasets, reaching state-of-the-art accuracy of on Flickr30k Entities and on ReferItGame dataset. Figure 6 (bottom) shows examples of improved phrase localization.
We propose the Multimodal Compact Bilinear Pooling (MCB) to combine visual and text representations. For visual question answering, our architecture with attention and multiple MCBs gives significant improvements on two VQA datasets compared to state-of-the-art. In the visual grounding task, introducing MCB pooling leads to improved phrase localization accuracy, indicating better interaction between query phrase representations and visual representations of proposal bounding boxes. The code to replicate our experiments is available at https://github.com/akirafukui/vqa-mcb.
We would like to thank Yang Gao and Oscar Beijbom for helpful discussions about Compact Bilinear Pooling. This work was supported by DARPA, AFRL, DoD MURI award N000141110688, NSF awards IIS-1427425 and IIS-1212798, and the Berkeley Artificial Intelligence Research (BAIR) Lab.
Proceedings of the International Conference on Machine Learning (ICML).
Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).