Complete 3D Scene Parsing from Single RGBD Image

by   Chuhang Zou, et al.

Inferring the location, shape, and class of each object in a single image is an important task in computer vision. In this paper, we aim to predict the full 3D parse of both visible and occluded portions of the scene from one RGBD image. We parse the scene by modeling objects as detailed CAD models with class labels and layouts as 3D planes. Such an interpretation is useful for visual reasoning and robotics, but difficult to produce due to the high degree of occlusion and the diversity of object classes. We follow the recent approaches that retrieve shape candidates for each RGBD region proposal, transfer and align associated 3D models to compose a scene that is consistent with observations. We propose to use support inference to aid interpretation and propose a retrieval scheme that uses convolutional neural networks (CNNs) to classify regions and retrieve objects with similar shapes. We demonstrate the performance of our method compared with the state-of-the-art on our new NYUd v2 dataset annotations which are semi-automatically labelled with detailed 3D shapes for all the objects.


page 2

page 4

page 6

page 11

page 12


Predicting Complete 3D Models of Indoor Scenes

One major goal of vision is to infer physical models of objects, surface...

Neural Implicit 3D Shapes from Single Images with Spatial Patterns

3D shape reconstruction from a single image has been a long-standing pro...

Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image

3D perception of object shapes from RGB image input is fundamental towar...

Peeking Behind Objects: Layered Depth Prediction from a Single Image

While conventional depth estimation can infer the geometry of a scene fr...

Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing

Monocular 3D object parsing is highly desirable in various scenarios inc...

Inferring Occluded Geometry Improves Performance when Retrieving an Object from Dense Clutter

Object search -- the problem of finding a target object in a cluttered s...

Parsing Geometry Using Structure-Aware Shape Templates

Real-life man-made objects often exhibit strong and easily-identifiable ...

Please sign up or login with your details

Forgot password? Click here to reset