Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images

by   Florian Kluger, et al.

Humans perceive and construct the surrounding world as an arrangement of simple parametric models. In particular, man-made environments commonly consist of volumetric primitives such as cuboids or cylinders. Inferring these primitives is an important step to attain high-level, abstract scene descriptions. Previous approaches directly estimate shape parameters from a 2D or 3D input, and are only able to reproduce simple objects, yet unable to accurately parse more complex 3D scenes. In contrast, we propose a robust estimator for primitive fitting, which can meaningfully abstract real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to 3D features, such as a depth map. We condition the network on previously detected parts of the scene, thus parsing it one-by-one. To obtain 3D features from a single RGB image, we additionally optimise a feature extraction CNN in an end-to-end manner. However, naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene behind. We thus propose an occlusion-aware distance metric correctly handling opaque scenes. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the challenging NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.


page 1

page 3

page 4

page 8

page 10

page 12

page 13

page 14


Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives

Given a set of calibrated images of a scene, we present an approach that...

Convex Decomposition of Indoor Scenes

We describe a method to parse a complex, cluttered indoor scene into pri...

View-volume Network for Semantic Scene Completion from a Single Depth Image

We introduce a View-Volume convolutional neural network (VVNet) for infe...

Single Image 3D Object Estimation with Primitive Graph Networks

Reconstructing 3D object from a single image (RGB or depth) is a fundame...

Primitive Fitting Using Deep Boundary Aware Geometric Segmentation

To identify and fit geometric primitives (e.g., planes, spheres, cylinde...

3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks

The success of various applications including robotics, digital content ...

Segmentation and Recovery of Superquadric Models using Convolutional Neural Networks

In this paper we address the problem of representing 3D visual data with...

Please sign up or login with your details

Forgot password? Click here to reset