DeepAI AI Chat
Log In Sign Up

Learning Visual Question Answering by Bootstrapping Hard Attention

08/01/2018
by   Mateusz Malinowski, et al.
6

Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while entirely ignoring some features. Even though the hard attention mechanism is thought to be non-differentiable, we found that the feature magnitudes correlate with semantic relevance, and provide a useful signal for our mechanism's attentional selection criterion. Because hard attention selects important features of the input information, it can also be more efficient than analogous soft attention mechanisms. This is especially important for recent approaches that use non-local pairwise operations, whereby computational and memory costs are quadratic in the size of the set of features.

READ FULL TEXT

page 2

page 14

page 15

09/05/2019

A Better Way to Attend: Attention with Trees for Video Question Answering

We propose a new attention model for video question answering. The main ...
05/11/2018

Reciprocal Attention Fusion for Visual Question Answering

Existing attention mechanisms either attend to local image grid or objec...
08/09/2019

Question-Agnostic Attention for Visual Question Answering

Visual Question Answering (VQA) models employ attention mechanisms to di...
05/24/2016

Hierarchical Memory Networks

Memory networks are neural networks with an explicit memory component th...
03/25/2018

Pay More Attention - Neural Architectures for Question-Answering

Machine comprehension is a representative task of natural language under...
10/22/2018

A Fully Attention-Based Information Retriever

Recurrent neural networks are now the state-of-the-art in natural langua...
05/05/2021

Soft-Attention Improves Skin Cancer Classification Performance

In clinical applications, neural networks must focus on and highlight th...

Code Repositories

PyTorch-AdaHAN

An unofficial PyTorch implementation of the HAN and AdaHAN models presented in the "Learning Visual Question Answering by Bootstrapping Hard Attention" research paper.


view repo