Vision Transformer for Small-Size Datasets

12/27/2021

∙

Recently, the Vision Transformer (ViT), which applied the transformer structure to the image classification task, has outperformed convolutional neural networks. However, the high performance of the ViT results from pre-training using a large-size dataset such as JFT-300M, and its dependence on a large dataset is interpreted as due to low locality inductive bias. This paper proposes Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA), which effectively solve the lack of locality inductive bias and enable it to learn from scratch even on small-size datasets. Moreover, SPT and LSA are generic and effective add-on modules that are easily applicable to various ViTs. Experimental results show that when both SPT and LSA were applied to the ViTs, the performance improved by an average of 2.96 is a representative small-size dataset. Especially, Swin Transformer achieved an overwhelming performance improvement of 4.08 LSA.

READ FULL TEXT

Vision Transformer for Small-Size Datasets

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

ExMobileViT: Lightweight Classifier Extension for Mobile Vision Transformer

Enhancing Performance of Vision Transformers on Small Datasets through Local Inductive Bias Incorporation

ViR:the Vision Reservoir

Vision Transformers provably learn spatial structure

Studying inductive biases in image classification task

CNN-based Local Vision Transformer for COVID-19 Diagnosis

Vision Transformer for Small-Size Datasets

Related Research

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

ExMobileViT: Lightweight Classifier Extension for Mobile Vision Transformer

Enhancing Performance of Vision Transformers on Small Datasets through Local Inductive Bias Incorporation

ViR:the Vision Reservoir

Vision Transformers provably learn spatial structure

Studying inductive biases in image classification task

CNN-based Local Vision Transformer for COVID-19 Diagnosis