Active learning for deep learning
We present an empirical study of active learning for Visual Question Answering, where a deep VQA model selects informative question-image pairs from a pool and queries an oracle for answers to maximally improve its performance under a limited query budget. Drawing analogies from human learning, we explore cramming (entropy), curiosity-driven (expected model change), and goal-driven (expected error reduction) active learning approaches, and propose a fast and effective goal-driven active learning scoring function to pick question-image pairs for deep VQA models under the Bayesian Neural Network framework. We find that deep VQA models need large amounts of training data before they can start asking informative questions. But once they do, all three approaches outperform the random selection baseline and achieve significant query savings. For the scenario where the model is allowed to ask generic questions about images but is evaluated only on specific questions (e.g., questions whose answer is either yes or no), our proposed goal-driven scoring function performs the best.READ FULL TEXT VIEW PDF
We introduce an interactive learning framework for the development and
Typical active learning strategies are designed for tasks, such as
Visual question answering (VQA) is a task of answering a visual question...
The study of algorithms to automatically answer visual questions current...
Despite their importance in training artificial intelligence systems, la...
Many pairwise classification tasks, such as paraphrase detection and
In the large-scale multiclass setting, assigning labels often consists o...
Active learning for deep learning
Visual Question Answering (VQA) [1, 8, 9, 11, 23, 28] is the task of taking in an image and a free-form natural language question and automatically answering the question. Correctly answering VQA questions arguably demonstrates machines’ image understanding, language understanding and and perhaps even some commonsense reasoning abilities. Previous works have demonstrated that deep models which combine image, question and answer representations, and are trained on large corpora of VQA data are effective at the VQA task.
Although such deep models are often deemed data-hungry, the flip-side is that their performance scales well with more training data. In Fig.1 we plot performance versus training set size of two representative deep VQA models: LSTM+CNN  and HieCoAtt  trained on random subsets of the VQA v1.0 dataset . We see that for both methods, accuracy improves significantly – by 12% – with every order of magnitude of more training data. As performance improvements still seem linear, it is reasonable to expect another 12% increase by collecting a VQA dataset 10 times larger. Such trends are invariant to choice of image feature 
and is also observed in ImageNet image classification. Note that improvements brought by additional training data may be orthogonal to improvements in model architecture.
However, collecting large quantities of annotated data is expensive. Even worse, as a result of long tail distributions, it will likely result in redundant questions and answers while still having insufficient training data for rare concepts. This is especially important for learning commonsense knowledge, as it is well known that humans tend to talk about unusual circumstances more often than commonsense knowledge which can be boring to talk about . Active learning helps address these issues. In active learning, a model is first trained on an initial training set. It then iteratively expands its training set by selecting potentially informative examples according to a query strategy, and seeking annotations on these examples. Previous works have shown that a carefully designed query strategy effectively reduces annotation effort required in a variety of tasks for shallow models. For deep models however, active learning literature is scarce and mainly focuses on classical unimodal tasks such as image and text classification.
In this work we study active learning for deep VQA models. VQA poses several unique challenges and opportunities for active learning.
First, VQA is a multimodal problem. Deep VQA models may combine Multi-Layer Perceptrons (MLPs), Convolutional Neural Nets (CNNs), Recurrent Neural Nets (RNNs) and even attention mechanisms to solve VQA. Such models are much more complex than MLPs or CNNs alone studied in existing active learning literature and need tailored query strategies.
Second, VQA questions are free-form and open-ended. In fact, VQA can play several roles from answering any generic question about an image, to answering only specific question types (e.g., questions with “yes/no” answers, or counting questions), to being a submodule in some other task (e.g., image captioning as in ). Each of these different scenarios may require a different active learning approach.
Finally, VQA can be thought of as a Visual Turing Test 
for computer vision systems. To answer questions such as “does this person have 20/20 vision” and “will the cat be able to jump onto the shelf”, the computer not only needs to understand the surface meaning of the image and the question, but it also needs to have sufficient commonsense knowledge about our world.One could argue that proposing informative questions about images is also a test of commonsense knowledge and intelligence.
We draw coarse analogies to human learning and explore three types of information-theoretic active learning query strategies:
Cramming – maximizing information gain in the training domain. The objective of this strategy is to efficiently memorize knowledge in an unlabeled pool of examples. This strategy selects unlabeled examples whose label the model is most uncertain about (maximum entropy).
Curiosity-driven learning – maximizing information gain in model space. The objective of this strategy is to select examples that could potentially change the belief on the model’s parameters (also known as expected model change). There might exist examples in the pool whose labels have high uncertainty but the model does not have enough capacity to capture them. In curiosity-driven learning the model will skip these examples. BALD [7, 13] is one such strategy for deep models, where examples are selected to maximize the reduction in entropy over model parameter space.
Goal-driven learning – maximizing information gain in the target domain. The objective of this strategy is to gather knowledge to better achieve a particular goal (also known as expected error reduction). To give an example from image classification, if the goal is to recognize digits i.e., the target domain is digit classification, dog images in the unlabeled pool are not relevant even though their labels might be uncertain or might change model parameters significantly. On the other hand, in addition to digit labels, some other non-digit labels such as the orientation of the image might be useful to the digit classification task. We propose a novel goal-driven query strategy that computes mutual information between pool questions and test questions under the Bayesian Neural Network [2, 6] framework.
We evaluate active learning performance on VQA v1.0  and v2.0  under the pool-based active learning setting described in Section 3. We show that active learning for deep VQA models requires a large amount of initial training data before they can achieve better scaling than random selection. In other words, the model needs to have enough knowledge to ask informative questions. But once it does, all three querying strategies outperform the random selection baseline, saving 27.3% and 19.0% answer annotation effort for VQA v1.0 and v2.0 respectively. Moreover, when the target task is restricted to answering only “yes/no” questions, our proposed goal-driven query strategy beats random selection and achieves the best performance out of the three active query strategies.
often rely on specific model simplifications and closed-form solutions. Deep neural networks however, are inherently complex non-linear functions. This poses challenges on uncertainty estimation.
In the context of deep active learning for language or image understanding, 
develops a margin-based query strategy on Restricted Boltzmann Machines for review sentiment classification. queries high-confidence web images for active fine-grained image classification.  proposes a query strategy based on feature space covering, applied to deep image classification. Closest to our work,  studies BALD , an expected model change query strategy computed under the Bayesian Neural Network [2, 6] framework applied to image classification.
In this work we study active learning for VQA. VQA is a challenging multimodal problem. Today’s state-of-the-art VQA models are deep neural networks. We take an information-theoretic perspective and study three active learning objectives: minimizing entropy in training domain (entropy), model space (expected model change) or target domain (expected error reduction). Drawing coarse analogy from human learning, we call them cramming, curiosity-driven and goal-driven learning respectively. We apply the Bayesian Neural Network [2, 6] framework to compute these strategies. In particular, for goal-driven learning which was deemed impractical beyond binary classification on small datasets , we propose a fast and effective query scoring function that speeds up computation by hundreds of millions of times, and show that it is effective for VQA which has classes and examples on contemporary multi-modal deep neural nets.
Building machines that demonstrate curiosity – machines that improve themselves through conversations with humans – is an important problem in AI.
uses reinforcement learning to learn an agent that plays a “Guess What?” game: finding out which object in the image the user is looking at by asking questions.  studies grounded visual dialog 
between two machines in collaborative image retrieval, where one machine as the “answerer” has an image and answers questions about the image while the other as “questioner” asks questions to retrieve the image at the end of the conversation. Both machines are learnt to better perform the task using reinforcement learning.
In this work we study visual “conversations” from an active learning perspective. In each round of the conversation, a VQA model strategically chooses an informative question about an image and queries an oracle to get an answer. Each round of “conversation” provides a new VQA training example which improves the VQA model.
We study a pool-based active learning setting for VQA: A VQA model is first trained on an initial training set . It then iteratively grows by greedily selecting batches of high-scoring question-image pairs from a human-curated pool according to a query scoring function . The selected pairs are sent to an oracle for one of ground truth answers , and tuples are added as new examples to . 111VQA models require a large training set to be effective. To avoid prohibitive data collection cost and focus on evaluating active learning query strategies, in this work we study pool-based active learning which makes use of existing VQA datasets. Having the model select or even generate questions for images it would liked answered, as opposed to picking from a pool of pairs is a direction for future research.
We take an information-theoretic perspective and explore cramming, curiosity-driven, and goal-driven query strategies as described in Section 1. However computing for those query strategies directly is intractable, as they require taking expectations under the model parameter distribution. So in Section 3.1 we first introduce a Bayesian VQA model which enables variational approximation of the model parameter distribution. And then Section 3.2 introduces the query scoring functions and their approximations.
We start with the LSTM+CNN VQA model 
. The model encodes an image into a feature vector using the VGG-net
CNN, encodes a question into a feature vector by learning a Long Short Term Memory (LSTM) RNN, and then learns a multi-layer perceptron on top that combines the image feature and the question feature to predict a probabilistic distribution over topmost common answers.
In order to learn a variational approximation of the posterior model distribution, we adopt the Bayesian Neural Network framework [2, 6] and introduce a Bayesian LSTM+CNN model for VQA. Let be the parameters of the LSTM and the multi-layer perceptron (we use a frozen pre-trained CNN). We learn a weight-generating model with parameter :
Let be the probabilistic distribution of weights generated by this model. Following [2, 6], we learn by minimizing KL divergence so serves as a variational approximation to the true model parameter posterior . Specifically we minimize
using batch Stochastic Gradient Descent (SGD) to learn. In practice, can be naively approxmiated with a parametric hybrid - norm . Experiments show that such an naive approximation does not have a significant impact on active learning results. So in experiments we set this term to 0. How to come up with a more informative prior term is an open problem for Bayesian Neural Networks.
Let be the predicted -dimensional answer distribution of the VQA model for question-image pair when using as model parameters. A Bayesian VQA prediction for using variational approximation is therefore given by:
We experiment with 3 active learning query strategies: cramming, curiosity-driven learning and goal-driven learning.
Cramming or “uncertainty sampling”  minimizes uncertainty (entropy) of answers for questions in the pool. It selects whose answer ’s distribution has maximum entropy. This is a classical active learning approach commonly used in practice.
Curiosity-driven learning or “expected model change” minimizes uncertainty (entropy) of model parameter distribution . It selects whose answer would expectedly bring steepest decrease in model parameter entropy if added to the training set.
Intuitively, computes the divergence of answer predictions under different model parameters. If plausible models are making divergent predictions on a question-image pair , the answer to this would rule out many of those models and thereby reduce confusion.
Goal-driven learning or “expected error reduction” minimizes uncertainty (entropy) on answers to a given set of unlabeled test question-image pairs , against which the model will be evaluated. The goal-driven query strategy selects the pool question-image pair that has the maximum total mutual information with . That is, it queries pairs which maximize:
For term , observe that when the model parameter is given, and are two different VQA questions so their answers – and respectively – are predicted independently. In other words, and are independent conditioned on . Therefore we can take expectation over model parameter
to compute this joint probability term:
Let be the number of samples of , be the number of possible answers, and be the number of examples in the pool. Computing for all examples in the pool following Eq. 3.2 has a time complexity of . For VQA typically the pool and test corpora each contains hundreds of thousands of examples and there are 1000 possible answers, e.g., , and . We typically use samples in our experiments. So computing Eq. 3.2 is still time-consuming and can be prohibitive for even larger VQA datasets. To speed up computation, we approximate using first-order Taylor expansion and discover that the following approximation holds empirically (more details can be found in Appendix A and B):
Eq. 3.2 brings drastic improvements to time complexity. It can be computed as a dot-product between two vectors of length . One only involves pool questions . The other one only involves test questions and can be precomputed for all pool questions. Precomputing vectors for test questions has a time complexity of . Computing Eq. 3.2 using the precomputed test vector has a time complexity of . So the overall time complexity is linear to dataset size and the number of possible answers .
In previous works, goal-driven learning was deemed impractical beyond binary classification on small datasets 29]12] and Gaussian Process . However on VQA, computing such scoring functions would require learning a new set of model parameters for every possible combinations of and then making predictions on all using the learnt model. That would require forward-backward passes (10 billion epochs) for VQA neural nets. Instead our Monte-Carlo approximation of Eq. 3.2 only involves making predictions on and , and avoids training new models for each of answers when computing . In our approach, the operation with the highest time complexity is a matrix multiplication operation which in practice is not the bottleneck. The most time-consuming operation – computing scores for and – costs approximately forward passes (75 epochs), a speed up of more than times. Our approach is easily parallelizable and works for all Bayesian Neural Networks.
Our active learning procedure is summarized in Algorithm 1.
We evaluate cramming (entropy), curiosity-driven and goal-driven active learning strategies against passive learning on the VQA v1.0  and v2.0  datasets. The VQA v1.0 dataset consists of 614,163 VQA questions with human answers on 204,721 COCO  images. The VQA v2.0 dataset augments the VQA v1.0 dataset and brings dataset balancing: every question in VQA v2.0 is paired with two similar images that have different answers to the question. So VQA v2.0 doubles the amount of data and models need to focus on the image to do well on VQA v2.0.
We choose a random initial training set of pairs from the TRAIN split, use the rest of TRAIN as pool and report VQA accuracy  on the VAL split. We run the active learning loop for iterations. We sample model parameter for times for query score computation. For passive learning i.e. querying pairs randomly, we set . In each iteration pairs are added to , resulting in a training set of examples by the end of iteration 50.
For VQA model, we use the Bayesian LSTM+CNN model described in Section 3.1. In every active learning iteration we train the model for epochs with learning rate and batch size for learning .
Fig. 2 (left), (right) show the active learning results on VQA v1.0 and v2.0 respectively. On both datasets, all 3 active learning methods perform similarly and all of them outperform passive learning. On VQA v1.0, passive learning queries 88k answers before reaching 51% accuracy, where as all active learning methods need only 64k queries, achieving a saving of 27.3%. It shows that active learning is able to effectively tell informative VQA questions from redundant questions, even among high-quality questions generated by humans. Similarly at 46% accuracy, active learning on VQA v2.0 achieves a saving of 19.0%. Savings on VQA v2.0 is lower, possibly because dataset balancing in VQA v2.0 improves the informativeness of even a random example.
Table 1 shows that for each pair of active learning methods, what percentage of the query pairs are selected by both methods on VQA v2.0 (overlap between their training sets). For the VQA task, active learning methods seem to agree on which pairs are more informative. They have more than 80% of pairs in common, while against passive learning they only share 27 % of pairs.
On VQA v2.0, we also experiment with smaller initial training sets to study the impact of training set size on active learning performance. Fig. 3 shows the results. For all initial training set sizes, the breakpoint when active learning methods start to outperform passive learning is around to examples. It shows that active learning methods do require a large training set size to start asking informative questions. Models with smaller initial training set sizes tend to show less and inconsistent data savings compared to , possibly because such models are less capable of telling informative questions from redundant ones. In addition, entropy shows fluctuating performance while curiosity-driven learning performs consistantly better than both entropy and passive learning irrespective of initial training set size.
To evaluate our goal-driven learning approach, we keep the initial training set and the pool unchanged for VQA v2.0 – the model is allowed to ask all kinds of questions from the VQA v2.0 TRAIN split – but will be evaluated on only “yes/no” questions (questions whose answers are “yes” or “no” ) in the VAL split. This task tests our proposed goal-driven active learning approach’s ability to focus on achieving the goal of answering “yes/no” questions more accurately.
Fig. 4 (top) shows the performance of active and passive learning approaches on this task.333We also found that updating from previous iteration in Algorithm 1 step 10 leads to slight overfitting that affects mutual information approximation. So for this task, is initialized from scratch in every iteration. Our goal-driven active learning approach is able to select relevant questions as queries and outperforms passive learning. Curiosity-driven and entropy approaches perform poorly. They are not aware of the task and tend to be attracted to harder, open-ended questions, which are not very relevant to the task.
Fig. 4 (bottom) shows a closer examination of the composition of questions queried by the goal-driven learning approach compared to baseline approaches. The goal-driven learning approach queries mostly “yes/no” questions, which are presumably more useful for the task. Note that the approach was not told that the downstream task is to answer “yes/no” questions. The approach figures out which questions will be informative to ask just based on samples from the downstream task. It shows that the goal-driven scoring function in Eq. 3.2, as well as the approximations in Eq. 3.2 are indeed effective for selecting informative questions.
As an “upper bound”, it is reasonable to assume444Note that this is not necessarily the case. Even non-yes/no questions can help a VQA model get better at answering yes/no questions by learning concepts from non-yes/no questions that can later come handy for yes/no questions. For example “Q: What is the man doing? A: Surfing” can be as useful as “Q: Is the man surfing? A: Yes”. that “yes/no” questions are more desirable for this task. Imagine a passive learning method that “cheats”: one that is aware that it will be tested only on “yes/no” questions, as well as knowing which questions are “yes/no’ questions in the pool, so it restricts itself to query only “yes/no” questions. How does our goal-driven learning approach compare with such a method that only learns from “yes/no” questions? Fig. 5 shows the results. Our goal-driven learning is able to compete with the “cheating” approach. In fact, of all 167,499 “yes/no” questions in the VQA v2.0 TRAIN split, goal-driven learning finds 38% of them by iteration 25, and 50% of them by iteration 50. That might also have made finding the remaining “yes/no” questions more difficult which explains the drop of the rate of “yes/no” question towards later iterations. We expect that a larger pool (i.e. a larger VQA dataset) would reduce these issues.
In this work we discussed three active learning strategies – cramming (entropy), curiosity-driven learning and goal-driven learning – for Visual Question Answering using deep multimodal neural networks. Our results show that deep VQA models require - training questions for active learning before they are able to ask informative questions and achieve better scaling than randomly selecting questions for labeling. Once the training set is large enough, several active learning strategies achieve significant savings in answer annotation cost. Our proposed goal-driven query strategy in particular, shows a significant advantage on improving performance when the downstream task involves answering a specific type of VQA questions.
Jointly selecting batches of examples as queries  and formulating active learning as a decision making problem  (greedily selecting the batch that reduces entropy by the most for the current iteration may not be the optimal decision) have been shown to improve optimality in active learning query selection. Combining those approaches with deep neural networks under the Bayesian Neural Network framework are promising future directions.
The pool-based active learning setup explored in this work selects unlabeled human generated question-image pairs and asks the oracle for answers. For building VQA datasets however, collecting human-generated questions paired with each image is also a substantial portion of the overall cost. Hence, starting from a bank of questions and an unaligned bank of images, and having the model decide which question it would like to pair with each image to use as a query would result in a further reduction in cost. Note that such a model would need to not only reason about the informativeness of a question-image pair, but also about the relevance of a question to the image [27, 22]. Evaluating such an approach would require collecting new VQA datasets with humans in the loop to give answers – which we show would require - answers before the model could start selecting informative images and questions. Going one step further, we could also envision a model that generates new questions rather than selecting from a pool of questions. That would require a generative model that can perform inference to optimize for the active learning objectives. We hope that our work serves as a foundation for these future research directions.
We thank Michael Cogswell and Qing Sun for discussions about the active learning strategies. This work was funded in part by an NSF CAREER award, ONR YIP award, Allen Distinguished Investigator award from the Paul G. Allen Family Foundation, Google Faculty Research Award, and Amazon Academic Research Award to DP. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1613–1622. PMLR, 2015.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.In Proceedings of the 33rd International Conference on Machine Learning (ICML), pages 1050–1059. PMLR, 2016.
Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI), volume 7, pages 823–829. AAAI, 2007.
Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 937–946. ACL, 2017.
Ask your neurons: A neural-based approach to answering questions about images.In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1–9. IEEE, 2015.
Systematic evaluation of convolution neural network advances on the ImageNet.Computer Vision and Image Understanding, 161:11–19, 2017.
Combining active learning and semi-supervised learning using gaussian fields and harmonic functions.In ICML 2003 workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining, pages 58–65, 2003.
In Section 3.2, we discuss our proposed goal-driven query strategy that minimizes uncertainty (entropy) on answers to a given set of unlabeled test question-image pairs , against which the model will be evaluated. It queries pairs which maximize:
Recall that we propose an approximation for term as follows:
Let us define four matrices as follows:
Here is an matrix, is a matrix, is an matrix and is a matrix. With we could rewrite Eq. A in matrix form:
|Rewriting in matrix form.|
|Sum of reduces to 1.|
|Property of trace.|
|Rewriting in probability form.|
As stated in Section 3.2, the above equation can be computed as a dot-product between two vectors of length . One vector is matrix expanded into a vector. It only involves pool questions . The other vector is expanded into a vector. It only involves test questions and it is shared for all pool questions , so it can be precomputed for all . Precomputing for test questions has a time complexity of . Note that is a diagonal matrix, so multiplying with only takes operations. In the same way, computing for all has a time complexity of . The time complexity of their dot product for all is merely . So the overall time complexity is . The overall time complexity is linear to both dataset size and and the number of possible answers , so our approach can easily scale to very large datasets and more VQA answers.
Our entropy, curiosity-driven and goal-driven scoring functions use 3 types of approximations
Variational distribution as approximation to model parameter distribution .
Monte Carlo sampling over for computing expectation over .
Fast approximation to mutual information in Eq. A.
For (a), since the space of model parameters is very large, it is intractable to evaluate how accurate substitutes for expectation computation. But nevertheless our goal-driven learning results in Section 4.3 suggest that Eq. 3.2 computed using is indeed useful for selecting relevant examples. It remains as an open problem that how to quantitatively evaluate the quality of for the purpose of uncertainty estimation and expectation computation.
For (b), we study the convergence patterns of Monte Carlo sampling. Specifically, given an arbitrary VQA model555For our experiments we use the model from curiosity-driven learning at iteration 50. This choice is made arbitrarily and does not change conclusions., we compute scores using Eq. 3.2 (entropy), 3.2 (curiosity-driven) and 3.2 (goal-driven) for 200 random examples from the pool using samples from , and compare them with in terms of rank correlation (Spearman’s ). Note that we use different seeds for the different values, i.e. samples for do not overlap with samples for . Fig. 6 shows the results. Entropy, curiosity-driven and goal-driven scoring functions require increasingly more samples of model parameters to converge in terms of ranking. To reach , entropy, curiosity-driven and goal-driven scoring functions require 10, 20 and 50 samples from respectively. Fig. 8 shows how the actual scores of examples change according to number of samples from for 50 random examples in the pool. The entropy and curiosity-driven scores seem to converge with a large number of samples. The goal-driven scores however, tend to first increase and then decrease with the number of samples and have not yet converged by samples, which is a limitation of the Monte Carlo sampling approach. Despite that, the relative rankings based on which the queries are selected have mostly converged. Upper- and lower-bounds of Eq. 3.2 that might improve convergence are opportunities for future research.
For (c), we plot goal-driven scores Eq. 3.2 as the x-axis versus our fast approximations Eq. 3.2 as the y-axis for 200 random examples from the pool using samples from . Because Eq. 3.2 does not scale well to large datasets, we use a subset of 200 random pairs from the VAL split as the test domain for both Eq. 3.2 and Eq. 3.2. Fig. 8 shows the results. Our fast approximations are mostly linear to the goal-driven scores. The slope changes according to the number of model parameter samples . That is probably because our approximation (see Section A for details) overestimates for . The rank correlations between goal-driven scores and their fast approximations remain high, e.g., above even for , which is sufficient for query selection.