Log In Sign Up

Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers

by   Katelyn Morrison, et al.

Recently, vision transformers and MLP-based models have been developed in order to address some of the prevalent weaknesses in convolutional neural networks. Due to the novelty of transformers being used in this domain along with the self-attention mechanism, it remains unclear to what degree these architectures are robust to corruptions. Despite some works proposing that data augmentation remains essential for a model to be robust against corruptions, we propose to explore the impact that the architecture has on corruption robustness. We find that vision transformer architectures are inherently more robust to corruptions than the ResNet-50 and MLP-Mixers. We also find that vision transformers with 5 times fewer parameters than a ResNet-50 have more shape bias. Our code is available to reproduce.


page 1

page 2

page 3

page 4


Can CNNs Be More Robust Than Transformers?

The recent success of Vision Transformers is shaking the long dominance ...

Robustness Verification for Transformers

Robustness verification that aims to formally certify the prediction beh...

Are Vision Transformers Robust to Spurious Correlations?

Deep neural networks may be susceptible to learning spurious correlation...

Flowformer: Linearizing Transformers with Conservation Flows

Transformers based on the attention mechanism have achieved impressive s...

An Impartial Take to the CNN vs Transformer Robustness Contest

Following the surge of popularity of Transformers in Computer Vision, se...

Translational Equivariance in Kernelizable Attention

While Transformer architectures have show remarkable success, they are b...

Vision Xformers: Efficient Attention for Image Classification

Although transformers have become the neural architectures of choice for...

Code Repositories


We investigated corruption robustness across different architectures including Convolutional Neural Networks, Vision Transformers, and the MLP-Mixer.

view repo