Couplformer:Rethinking Vision Transformer with Coupling Attention Map

12/10/2021
by   Hai Lan, et al.
0

With the development of the self-attention mechanism, the Transformer model has demonstrated its outstanding performance in the computer vision domain. However, the massive computation brought from the full attention mechanism became a heavy burden for memory consumption. Sequentially, the limitation of memory reduces the possibility of improving the Transformer model. To remedy this problem, we propose a novel memory economy attention mechanism named Couplformer, which decouples the attention map into two sub-matrices and generates the alignment scores from spatial information. A series of different scale image classification tasks are applied to evaluate the effectiveness of our model. The result of experiments shows that on the ImageNet-1k classification task, the Couplformer can significantly decrease 28 consumption compared with regular Transformer while accessing sufficient accuracy requirements and outperforming 0.92 the same memory footprint. As a result, the Couplformer can serve as an efficient backbone in visual tasks, and provide a novel perspective on the attention mechanism for researchers.

READ FULL TEXT
research
12/23/2020

A Survey on Visual Transformer

Transformer is a type of deep neural network mainly based on self-attent...
research
04/27/2022

Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

A long time ago in the machine learning literature, the idea of incorpor...
research
11/10/2021

Attention Approximates Sparse Distributed Memory

While Attention has come to be an important mechanism in deep learning, ...
research
07/18/2022

Multi-manifold Attention for Vision Transformers

Vision Transformer are very popular nowadays due to their state-of-the-a...
research
07/15/2019

Agglomerative Attention

Neural networks using transformer-based architectures have recently demo...
research
01/26/2022

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

Attention mechanism has been widely believed as the key to success of vi...
research
09/29/2020

Attention that does not Explain Away

Models based on the Transformer architecture have achieved better accura...

Please sign up or login with your details

Forgot password? Click here to reset