FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

03/20/2022
by   Xuanyao Chen, et al.
4

Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (56.8 mAP) achieves on par performance with state-of-the-art 3D detection model CenterPoint (56.6 mAP) using a 32-beam LiDAR.

READ FULL TEXT

page 8

page 12

page 13

page 14

research
04/19/2021

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

How should representations from complementary sensors be integrated for ...
research
12/01/2021

Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles

In autonomous driving, there has been an explosion in the use of deep ne...
research
02/22/2023

Transformer-Based Sensor Fusion for Autonomous Driving: A Survey

Sensor fusion is an essential topic in many perception systems, such as ...
research
04/28/2023

Fusion is Not Enough: Single-Modal Attacks to Compromise Fusion Models in Autonomous Driving

Multi-sensor fusion (MSF) is widely adopted for perception in autonomous...
research
02/08/2020

Multi-Modality Cascaded Fusion Technology for Autonomous Driving

Multi-modality fusion is the guarantee of the stability of autonomous dr...
research
01/11/2022

End-To-End Optimization of LiDAR Beam Configuration for 3D Object Detection and Localization

Existing learning methods for LiDAR-based applications use 3D points sca...
research
09/14/2021

Detecting Safety Problems of Multi-Sensor Fusion in Autonomous Driving

Autonomous driving (AD) systems have been thriving in recent years. In g...

Please sign up or login with your details

Forgot password? Click here to reset