Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles

12/01/2021
by   Sammy Sidhu, et al.
0

In autonomous driving, there has been an explosion in the use of deep neural networks for perception, prediction and planning tasks. As autonomous vehicles (AVs) move closer to production, multi-modal sensor inputs and heterogeneous vehicle fleets with different sets of sensor platforms are becoming increasingly common in the industry. However, neural network architectures typically target specific sensor platforms and are not robust to changes in input, making the problem of scaling and model deployment particularly difficult. Furthermore, most players still treat the problem of optimizing software and hardware as entirely independent problems. We propose a new end to end architecture, Generalized Sensor Fusion (GSF), which is designed in such a way that both sensor inputs and target tasks are modular and modifiable. This enables AV system designers to easily experiment with different sensor configurations and methods and opens up the ability to deploy on heterogeneous fleets using the same models that are shared across a large engineering organization. Using this system, we report experimental results where we demonstrate near-parity of an expensive high-density (HD) LiDAR sensor with a cheap low-density (LD) LiDAR plus camera setup in the 3D object detection task. This paves the way for the industry to jointly design hardware and software architectures as well as large fleets with heterogeneous configurations.

READ FULL TEXT

page 5

page 9

research
03/20/2022

FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

Sensor fusion is an essential topic in many perception systems, such as ...
research
06/23/2020

Towards Robust Sensor Fusion in Visual Perception

We study the problem of robust sensor fusion in visual perception, espec...
research
05/11/2023

DeepSTEP – Deep Learning-Based Spatio-Temporal End-To-End Perception for Autonomous Vehicles

Autonomous vehicles demand high accuracy and robustness of perception al...
research
03/13/2023

A Modular Platform For Collaborative, Distributed Sensor Fusion

Leading autonomous vehicle (AV) platforms and testing infrastructures ar...
research
06/13/2021

Security Analysis of Camera-LiDAR Semantic-Level Fusion Against Black-Box Attacks on Autonomous Vehicles

To enable safe and reliable decision-making, autonomous vehicles (AVs) f...
research
12/22/2014

Multi-modal Sensor Registration for Vehicle Perception via Deep Neural Networks

The ability to simultaneously leverage multiple modes of sensor informat...
research
04/08/2021

Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems

Optimizing the quality of result (QoR) and the quality of service (QoS) ...

Please sign up or login with your details

Forgot password? Click here to reset