DeepAI AI Chat
Log In Sign Up

Efficiency 360: Efficient Vision Transformers

02/16/2023
by   Badri N. Patro, et al.
Microsoft
5

Transformers are widely used for solving tasks in natural language processing, computer vision, speech, and music domains. In this paper, we talk about the efficiency of transformers in terms of memory (the number of parameters), computation cost (number of floating points operations), and performance of models, including accuracy, the robustness of the model, and fair & bias-free features. We mainly discuss the vision transformer for the image classification task. Our contribution is to introduce an efficient 360 framework, which includes various aspects of the vision transformer, to make it more efficient for industrial applications. By considering those applications, we categorize them into multiple dimensions such as privacy, robustness, transparency, fairness, inclusiveness, continual learning, probabilistic models, approximation, computational complexity, and spectral complexity. We compare various vision transformer models based on their performance, the number of parameters, and the number of floating point operations (FLOPs) on multiple datasets.

READ FULL TEXT
09/14/2020

Efficient Transformers: A Survey

Transformer model architectures have garnered immense interest lately du...
09/15/2022

Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision

Determining an appropriate number of attention heads on one hand and the...
06/30/2021

Improving the Efficiency of Transformers for Resource-Constrained Devices

Transformers provide promising accuracy and have become popular and used...
02/08/2023

Mitigating Bias in Visual Transformers via Targeted Alignment

As transformer architectures become increasingly prevalent in computer v...
06/15/2022

Efficient Adaptive Ensembling for Image Classification

In recent times, except for sporadic cases, the trend in Computer Vision...
12/03/2018

Structure Learning Using Forced Pruning

Markov networks are widely used in many Machine Learning applications in...
07/14/2022

Rethinking Attention Mechanism in Time Series Classification

Attention-based models have been widely used in many areas, such as comp...