DeepAI AI Chat
Log In Sign Up

Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

by   Derya Soydaner, et al.
KU Leuven

A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.


page 1

page 2

page 3

page 4


Couplformer:Rethinking Vision Transformer with Coupling Attention Map

With the development of the self-attention mechanism, the Transformer mo...

Text Classification with Lexicon from PreAttention Mechanism

A comprehensive and high-quality lexicon plays a crucial role in traditi...

Aggregated Sparse Attention for Steering Angle Prediction

In this paper, we apply the attention mechanism to autonomous driving fo...

Agglomerative Attention

Neural networks using transformer-based architectures have recently demo...

Superconnexivity Reconsidered

We reconsider the idea of superconnexivity, an idea that has not receive...

Staying True to Your Word: (How) Can Attention Become Explanation?

The attention mechanism has quickly become ubiquitous in NLP. In additio...

Attention-based Assisted Excitation for Salient Object Segmentation

Visual attention brings significant progress for Convolution Neural Netw...