Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

04/27/2022
by   Derya Soydaner, et al.
326

A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/10/2021

Couplformer:Rethinking Vision Transformer with Coupling Attention Map

With the development of the self-attention mechanism, the Transformer mo...
research
02/18/2020

Text Classification with Lexicon from PreAttention Mechanism

A comprehensive and high-quality lexicon plays a crucial role in traditi...
research
03/15/2018

Aggregated Sparse Attention for Steering Angle Prediction

In this paper, we apply the attention mechanism to autonomous driving fo...
research
07/15/2019

Agglomerative Attention

Neural networks using transformer-based architectures have recently demo...
research
04/14/2022

Superconnexivity Reconsidered

We reconsider the idea of superconnexivity, an idea that has not receive...
research
05/19/2020

Staying True to Your Word: (How) Can Attention Become Explanation?

The attention mechanism has quickly become ubiquitous in NLP. In additio...
research
08/30/2023

Denoising Attention for Query-aware User Modeling in Personalized Search

The personalization of search results has gained increasing attention in...

Please sign up or login with your details

Forgot password? Click here to reset