Visual Attention Methods in Deep Learning: An In-Depth Survey

04/16/2022
by   Mohammed Hassanin, et al.
0

Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated in one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey specific to attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and open questions related to attention mechanism in general. Finally, we recommend possible future research directions for deep attention.

READ FULL TEXT

page 14

page 16

research
03/27/2022

A General Survey on Attention Mechanisms in Deep Learning

Attention is an important mechanism that can be employed for a variety o...
research
03/03/2022

Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work

Vision Transformers (ViTs) are becoming more popular and dominating tech...
research
04/05/2019

An Attentive Survey of Attention Models

Attention Model has now become an important concept in neural networks t...
research
02/15/2022

The Quarks of Attention

Attention plays a fundamental role in both natural and artificial intell...
research
08/14/2018

Small Sample Learning in Big Data Era

As a promising area in artificial intelligence, a new learning paradigm,...
research
12/29/2019

A Gentle Introduction to Deep Learning for Graphs

The adaptive processing of graph data is a long-standing research topic ...
research
06/17/2018

How Could Polyhedral Theory Harness Deep Learning?

The holy grail of deep learning is to come up with an automatic method t...

Please sign up or login with your details

Forgot password? Click here to reset