How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

06/18/2021
by   Andreas Steiner, et al.
0

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation (“AugReg” for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.

READ FULL TEXT
12/16/2021

How to augment your ViTs? Consistency loss and StyleAug, a random style transfer augmentation

The Vision Transformer (ViT) architecture has recently achieved competit...
09/08/2022

Video Vision Transformers for Violence Detection

Law enforcement and city safety are significantly impacted by detecting ...
05/03/2022

Better plain ViT baselines for ImageNet-1k

It is commonly accepted that the Vision Transformer model requires sophi...
10/05/2020

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?

Task-agnostic forms of data augmentation have proven widely effective in...
01/31/2022

Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations

Existing techniques for model inversion typically rely on hard-to-tune r...
11/18/2021

TransMix: Attend to Mix for Vision Transformers

Mixup-based augmentation has been found to be effective for generalizing...
07/22/2022

Applying Spatiotemporal Attention to Identify Distracted and Drowsy Driving with Vision Transformers

A 20 result of increased distraction and drowsiness. Drowsy and distract...

Code Repositories

download_augreg

Download ViT AugReg model weights from https://arxiv.org/abs/2106.10270


view repo