From Points to Multi-Object 3D Reconstruction

12/21/2020
by   Francis Engelmann, et al.
37

We propose a method to detect and reconstruct multiple 3D objects from a single RGB image. The key idea is to optimize for detection, alignment and shape jointly over all objects in the RGB image, while focusing on realistic and physically plausible reconstructions. To this end, we propose a keypoint detector that localizes objects as center points and directly predicts all object properties, including 9-DoF bounding boxes and 3D shapes – all in a single forward pass. The proposed method formulates 3D shape reconstruction as a shape selection problem, i.e. it selects among exemplar shapes from a given database. This makes it agnostic to shape representations, which enables a lightweight reconstruction of realistic and visually-pleasing shapes based on CAD-models, while the training objective is formulated around point clouds and voxel representations. A collision-loss promotes non-intersecting objects, further increasing the reconstruction realism. Given the RGB image, the presented approach performs lightweight reconstruction in a single-stage, it is real-time capable, fully differentiable and end-to-end trainable. Our experiments compare multiple approaches for 9-DoF bounding box estimation, evaluate the novel shape-selection mechanism and compare to recent methods in terms of 3D bounding box estimation and 3D shape reconstruction quality.

READ FULL TEXT

page 1

page 2

page 3

page 6

page 7

page 10

page 11

page 12

research
12/16/2019

PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points

Detecting 3D objects from a single RGB image is intrinsically ambiguous,...
research
04/02/2020

DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes

We propose DOPS, a fast single-stage 3D object detection method for LIDA...
research
04/16/2019

Objects as Points

Detection identifies objects as axis-aligned boxes in an image. Most suc...
research
11/23/2016

Straight to Shapes: Real-time Detection of Encoded Shapes

Current object detection approaches predict bounding boxes, but these pr...
research
11/30/2016

Deep Cuboid Detection: Beyond 2D Bounding Boxes

We present a Deep Cuboid Detector which takes a consumer-quality RGB ima...
research
10/05/2021

3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video

3D object reconstructions of transparent and concave structured objects,...
research
11/18/2020

Diverse Plausible Shape Completions from Ambiguous Depth Images

We propose PSSNet, a network architecture for generating diverse plausib...

Please sign up or login with your details

Forgot password? Click here to reset