Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud

03/23/2019
by   Xinshuo Weng, et al.
0

Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor such as a depth camera, a stereo camera or LiDAR. On the other hand, single image based methods have significantly worse performance, but rightly so, as there is little explicit depth information in a 2D image. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, which we call pseudo-LiDAR point cloud. Then we can train a LiDAR-based 3D detection network with our pseudo-LiDAR end-to-end. Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal. Then an oriented 3D bounding box is detected for each frustum. To handle the large amount of noise in the pseudo-LiDAR, we propose two innovations: (1) use a 2D-3D bounding box consistency constraint, adjusting the predicted 3D bounding box to have a high overlap with its corresponding 2D proposal after projecting onto the image; (2) use the instance mask instead of the bounding box as the representation of 2D proposals, in order to reduce the number of points not belonging to the object in the point cloud frustum. Through our evaluation on the KITTI benchmark, we achieve the top-ranked performance on both bird's eye view and 3D object detection among all monocular methods, effectively quadrupling the performance over previous state-of-the-art.

READ FULL TEXT

page 3

page 4

page 5

page 7

research
04/21/2020

YOLO and K-Means Based 3D Object Detection Method on Image and Point Cloud

Lidar based 3D object detection and classification tasks are essential f...
research
09/11/2019

Multi-Sensor 3D Object Box Refinement for Autonomous Driving

We propose a 3D object detection system with multi-sensor refinement in ...
research
08/19/2019

BoxNet: A Deep Learning Method for 2D Bounding Box Estimation from Bird's-Eye View Point Cloud

We present a learning-based method to estimate the object bounding box f...
research
12/10/2019

Learning Depth-Guided Convolutions for Monocular 3D Object Detection

3D object detection from a single image without LiDAR is a challenging t...
research
04/13/2021

OCM3D: Object-Centric Monocular 3D Object Detection

Image-only and pseudo-LiDAR representations are commonly used for monocu...
research
11/24/2021

SM3D: Simultaneous Monocular Mapping and 3D Detection

Mapping and 3D detection are two major issues in vision-based robotics, ...
research
04/02/2019

Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction

We present MonoPSR, a monocular 3D object detection method that leverage...

Please sign up or login with your details

Forgot password? Click here to reset