DeepAI AI Chat
Log In Sign Up

Attention, please! A Critical Review of Neural Attention Models in Natural Language Processing

by   Andrea Galassi, et al.
University of Bologna
University of Modena and Reggio Emilia

Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area.


page 1

page 2

page 3

page 4


Exploring Different Dimensions of Attention for Uncertainty Detection

Neural networks with attention have proven effective for many natural la...

Attention Boosted Sequential Inference Model

Attention mechanism has been proven effective on natural language proces...

Transformadores: Fundamentos teoricos y Aplicaciones

Transformers are a neural network architecture originally designed for n...

Location Attention for Extrapolation to Longer Sequences

Neural networks are surprisingly good at interpolating and perform remar...

BERT: A Review of Applications in Natural Language Processing and Understanding

In this review, we describe the application of one of the most popular d...

Extending the Abstraction of Personality Types based on MBTI with Machine Learning and Natural Language Processing

A data-centric approach with Natural Language Processing (NLP) to predic...

Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer

Developing neural architectures that are capable of logical reasoning ha...