DeepAI AI Chat
Log In Sign Up

Learning Visual Question Answering by Bootstrapping Hard Attention

by   Mateusz Malinowski, et al.

Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while entirely ignoring some features. Even though the hard attention mechanism is thought to be non-differentiable, we found that the feature magnitudes correlate with semantic relevance, and provide a useful signal for our mechanism's attentional selection criterion. Because hard attention selects important features of the input information, it can also be more efficient than analogous soft attention mechanisms. This is especially important for recent approaches that use non-local pairwise operations, whereby computational and memory costs are quadratic in the size of the set of features.


page 2

page 14

page 15


A Better Way to Attend: Attention with Trees for Video Question Answering

We propose a new attention model for video question answering. The main ...

Reciprocal Attention Fusion for Visual Question Answering

Existing attention mechanisms either attend to local image grid or objec...

Question-Agnostic Attention for Visual Question Answering

Visual Question Answering (VQA) models employ attention mechanisms to di...

Hierarchical Memory Networks

Memory networks are neural networks with an explicit memory component th...

Pay More Attention - Neural Architectures for Question-Answering

Machine comprehension is a representative task of natural language under...

A Fully Attention-Based Information Retriever

Recurrent neural networks are now the state-of-the-art in natural langua...

Soft-Attention Improves Skin Cancer Classification Performance

In clinical applications, neural networks must focus on and highlight th...

Code Repositories


An unofficial PyTorch implementation of the HAN and AdaHAN models presented in the "Learning Visual Question Answering by Bootstrapping Hard Attention" research paper.

view repo