DeepAI AI Chat
Log In Sign Up

End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection

by   Rui Qian, et al.
cornell university
The Ohio State University

Reliable and accurate 3D object detection is a necessity for safe autonomous driving. Although LiDAR sensors can provide accurate 3D point cloud estimates of the environment, they are also prohibitively expensive for many settings. Recently, the introduction of pseudo-LiDAR (PL) has led to a drastic reduction in the accuracy gap between methods based on LiDAR sensors and those based on cheap stereo cameras. PL combines state-of-the-art deep neural networks for 3D depth estimation with those for 3D object detection by converting 2D depth map outputs to 3D point cloud inputs. However, so far these two networks have to be trained separately. In this paper, we introduce a new framework based on differentiable Change of Representation (CoR) modules that allow the entire PL pipeline to be trained end-to-end. The resulting framework is compatible with most state-of-the-art networks for both tasks and in combination with PointRCNN improves over PL consistently across all benchmarks – yielding the highest entry on the KITTI image-based 3D object detection leaderboard at the time of submission. Our code will be made available at


page 1

page 3

page 4

page 7

page 8

page 13

page 14


Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving

Detecting objects such as cars and pedestrians in 3D plays an indispensa...

PointPillars Backbone Type Selection For Fast and Accurate LiDAR Object Detection

3D object detection from LiDAR sensor data is an important topic in the ...

Virtual Sparse Convolution for Multimodal 3D Object Detection

Recently, virtual/pseudo-point-based 3D object detection that seamlessly...

Learning Depth-Guided Convolutions for Monocular 3D Object Detection

3D object detection from a single image without LiDAR is a challenging t...

SM3D: Simultaneous Monocular Mapping and 3D Detection

Mapping and 3D detection are two major issues in vision-based robotics, ...

VR3Dense: Voxel Representation Learning for 3D Object Detection and Monocular Dense Depth Reconstruction

3D object detection and dense depth estimation are one of the most vital...

Rethinking Pseudo-LiDAR Representation

The recently proposed pseudo-LiDAR based 3D detectors greatly improve th...

Code Repositories



view repo