Transformer-based models and hardware acceleration analysis in autonomous driving: A survey

04/21/2023
by   Juan Zhong, et al.
0

Transformer architectures have exhibited promising performance in various autonomous driving applications in recent years. On the other hand, its dedicated hardware acceleration on portable computational platforms has become the next critical step for practical deployment in real autonomous vehicles. This survey paper provides a comprehensive overview, benchmark, and analysis of Transformer-based models specifically tailored for autonomous driving tasks such as lane detection, segmentation, tracking, planning, and decision-making. We review different architectures for organizing Transformer inputs and outputs, such as encoder-decoder and encoder-only structures, and explore their respective advantages and disadvantages. Furthermore, we discuss Transformer-related operators and their hardware acceleration schemes in depth, taking into account key factors such as quantization and runtime. We specifically illustrate the operator level comparison between layers from convolutional neural network, Swin-Transformer, and Transformer with 4D encoder. The paper also highlights the challenges, trends, and current insights in Transformer-based models, addressing their hardware deployment and acceleration issues within the context of long-term autonomous driving applications.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/30/2020

Computing Systems for Autonomous Driving: State-of-the-Art and Challenges

The recent proliferation of computing technologies, e.g., sensors, compu...
research
09/05/2023

AutonomROS: A ReconROS-based Autonomonous Driving Unit

Autonomous driving has become an important research area in recent years...
research
07/18/2023

Light-Weight Vision Transformer with Parallel Local and Global Self-Attention

While transformer architectures have dominated computer vision in recent...
research
08/11/2023

Hardware Accelerators in Autonomous Driving

Computing platforms in autonomous vehicles record large amounts of data ...
research
10/10/2022

Edge Device Deployment of Multi-Tasking Network for Self-Driving Operations

A safe and robust autonomous driving system relies on accurate perceptio...
research
10/12/2021

LightSeq2: Accelerated Training for Transformer-based Models on GPUs

Transformer-based neural models are used in many AI applications. Traini...
research
03/18/2022

Laneformer: Object-aware Row-Column Transformers for Lane Detection

We present Laneformer, a conceptually simple yet powerful transformer-ba...

Please sign up or login with your details

Forgot password? Click here to reset