Going deeper with Image Transformers

03/31/2021
by   Hugo Touvron, et al.
0

Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of image transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two transformers architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for instance we obtain 86.3 Imagenet when training with no external data. Our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data.

READ FULL TEXT

page 19

page 20

research
05/21/2022

Vision Transformers in 2022: An Update on Tiny ImageNet

The recent advances in image transformers have shown impressive results ...
research
05/26/2021

Aggregating Nested Transformers

Although hierarchical structures are popular in recent vision transforme...
research
04/02/2021

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference

We design a family of image classification architectures that optimize t...
research
09/17/2014

Going Deeper with Convolutions

We propose a deep convolutional neural network architecture codenamed "I...
research
06/28/2021

Early Convolutions Help Transformers See Better

Vision transformer (ViT) models exhibit substandard optimizability. In p...
research
07/01/2023

More for Less: Compact Convolutional Transformers Enable Robust Medical Image Classification with Limited Data

Transformers are very powerful tools for a variety of tasks across domai...
research
01/25/2019

Equivariant Transformer Networks

How can prior knowledge on the transformation invariances of a domain be...

Please sign up or login with your details

Forgot password? Click here to reset