A Survey of Transformers

06/08/2021
by   Tianyang Lin, et al.
0

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/17/2022

Transformer for Graphs: An Overview from Architecture Perspective

Recently, Transformer model, which has achieved great success in many ar...
research
06/11/2023

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Transformer is a deep neural network that employs a self-attention mecha...
research
06/13/2022

Multimodal Learning with Transformers: A Survey

Transformer is a promising neural network learner, and has achieved grea...
research
02/03/2022

A Survey of Methods for Automated Algorithm Configuration

Algorithm configuration (AC) is concerned with the automated search of t...
research
06/07/2022

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

Transformers have achieved remarkable success in several domains, rangin...
research
02/15/2022

Transformers in Time Series: A Survey

Transformers have achieved superior performances in many tasks in natura...
research
02/22/2021

Position Information in Transformers: An Overview

Transformers are arguably the main workhorse in recent Natural Language ...

Please sign up or login with your details

Forgot password? Click here to reset