Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints

11/19/2021
by   Jaesin Ahn, et al.
0

A vision transformer (ViT) is the dominant model in the computer vision field. Despite numerous studies that mainly focus on dealing with inductive bias and complexity, there remains the problem of finding better transformer networks. For example, conventional transformer-based models usually use a projection layer for each query (Q), key (K), and value (V) embedding before multi-head self-attention. Insufficient consideration of semantic Q, K, and V embedding may lead to a performance drop. In this paper, we propose three types of structures for Q, K, and V embedding. The first structure utilizes two layers with ReLU, which is a non-linear embedding for Q, K, and V. The second involves sharing one of the non-linear layers to share knowledge among Q, K, and V. The third proposed structure shares all non-linear layers with code parameters. The codes are trainable, and the values determine the embedding process to be performed among Q, K, and V. Hence, we demonstrate the superior image classification performance of the proposed approaches in experiments compared to several state-of-the-art approaches. The proposed method achieved 71.4% with a few parameters (of 3.1M) on the ImageNet-1k dataset compared to that required by the original transformer model of XCiT-N12 (69.9%). Additionally, the method achieved 93.3% with only 2.9M parameters in transfer learning on average for the CIFAR-10, CIFAR-100, Stanford Cars datasets, and STL-10 datasets, which is better than the accuracy of 92.2% obtained via the original XCiT-N12 model.

READ FULL TEXT
research
09/29/2021

UFO-ViT: High Performance Linear Vision Transformer without Softmax

Vision transformers have become one of the most important models for com...
research
05/04/2022

Sequencer: Deep LSTM for Image Classification

In recent computer vision research, the advent of the Vision Transformer...
research
02/13/2022

BViT: Broad Attention based Vision Transformer

Recent works have demonstrated that transformer can achieve promising pe...
research
08/09/2021

RaftMLP: Do MLP-based Models Dream of Winning Over Computer Vision?

For the past ten years, CNN has reigned supreme in the world of computer...
research
03/14/2022

EIT: Efficiently Lead Inductive Biases to ViT

Vision Transformer (ViT) depends on properties similar to the inductive ...
research
11/05/2021

Hepatic vessel segmentation based on 3Dswin-transformer with inductive biased multi-head self-attention

Purpose: Segmentation of liver vessels from CT images is indispensable p...
research
09/04/2023

ExMobileViT: Lightweight Classifier Extension for Mobile Vision Transformer

The paper proposes an efficient structure for enhancing the performance ...

Please sign up or login with your details

Forgot password? Click here to reset