Attention, please! A survey of Neural Attention Models in Deep Learning

03/31/2021
by   Alana de Santana Correia, et al.
0

In humans, Attention is a core property of all perceptual and cognitive operations. Given our limited ability to process competing sources, attention mechanisms select, modulate, and focus on the information most relevant to behavior. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. For the last six years, this property has been widely explored in deep neural networks. Currently, the state-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models. We systematically reviewed hundreds of architectures in the area, identifying and discussing those in which attention has shown a significant impact. We also developed and made public an automated methodology to facilitate the development of reviews in the area. By critically analyzing 650 works, we describe the primary uses of attention in convolutional, recurrent networks and generative models, identifying common subgroups of uses and applications. Furthermore, we describe the impact of attention in different application domains and their impact on neural networks' interpretability. Finally, we list possible trends and opportunities for further research, hoping that this review will provide a succinct overview of the main attentional models in the area and guide researchers in developing future approaches that will drive further improvements.

READ FULL TEXT

page 4

page 13

page 27

research
04/05/2019

An Attentive Survey of Attention Models

Attention Model has now become an important concept in neural networks t...
research
12/11/2021

Neural Attention Models in Deep Learning: Survey and Taxonomy

Attention is a state of arousal capable of dealing with limited processi...
research
10/06/2021

Deep Neural Networks and Tabular Data: A Survey

Heterogeneous tabular data are the most commonly used form of data and a...
research
10/27/2016

A Review of 40 Years of Cognitive Architecture Research: Core Cognitive Abilities and Practical Applications

In this paper we present a broad overview of the last 40 years of resear...
research
04/07/2023

Attention: Marginal Probability is All You Need?

Attention mechanisms are a central property of cognitive systems allowin...
research
12/19/2022

An overview of open source Deep Learning-based libraries for Neuroscience

In recent years, deep learning revolutionized machine learning and its a...
research
06/25/2022

Data Augmentation techniques in time series domain: A survey and taxonomy

With the latest advances in deep learning generative models, it has not ...

Please sign up or login with your details

Forgot password? Click here to reset