Warping of Radar Data into Camera Image for Cross-Modal Supervision in Automotive Applications

12/23/2020
by   Christopher Grimm, et al.
7

In this paper, we present a novel framework to project automotive radar range-Doppler (RD) spectrum into camera image. The utilized warping operation is designed to be fully differentiable, which allows error backpropagation through the operation. This enables the training of neural networks (NN) operating exclusively on RD spectrum by utilizing labels provided from camera vision models. As the warping operation relies on accurate scene flow, additionally, we present a novel scene flow estimation algorithm fed from camera, lidar and radar, enabling us to improve the accuracy of the warping operation. We demonstrate the framework in multiple applications like direction-of-arrival (DoA) estimation, target detection, semantic segmentation and estimation of radar power from camera data. Extensive evaluations have been carried out for the DoA application and suggest superior quality for NN based estimators compared to classical estimators. The novel scene flow estimation approach is benchmarked against state-of-the-art scene flow algorithms and outperforms them by roughly a third.

READ FULL TEXT

page 6

page 10

page 11

page 12

page 13

page 14

page 15

page 17

research
03/02/2022

Self-Supervised Scene Flow Estimation with 4D Automotive Radar

Scene flow allows autonomous vehicles to reason about the arbitrary moti...
research
08/24/2021

Full-Velocity Radar Returns by Radar-Camera Fusion

A distinctive feature of Doppler radar is the measurement of velocity in...
research
04/02/2020

RSS-Net: Weakly-Supervised Multi-Class Semantic Segmentation with FMCW Radar

This paper presents an efficient annotation procedure and an application...
research
08/12/2023

4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion

Four-dimensional (4D) radar–visual odometry (4DRVO) integrates complemen...
research
10/04/2022

Implicit Warping for Animation with Image Sets

We present a new implicit warping framework for image animation using se...
research
05/04/2020

How to Train Your Dragon: Tamed Warping Network for Semantic Video Segmentation

Real-time semantic segmentation on high-resolution videos is challenging...
research
03/28/2023

F^2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

This paper presents a novel grid-based NeRF called F2-NeRF (Fast-Free-Ne...

Please sign up or login with your details

Forgot password? Click here to reset