Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation

04/02/2021
by   Karl Stelzner, et al.
11

We present ObSuRF, a method which turns a single image of a scene into a 3D model represented as a set of Neural Radiance Fields (NeRFs), with each NeRF corresponding to a different object. A single forward pass of an encoder network outputs a set of latent vectors describing the objects in the scene. These vectors are used independently to condition a NeRF decoder, defining the geometry and appearance of each object. We make learning more computationally efficient by deriving a novel loss, which allows training NeRFs on RGB-D inputs without explicit ray marching. After confirming that the model performs equal or better than state of the art on three 2D image segmentation benchmarks, we apply it to two multi-object 3D datasets: A multiview version of CLEVR, and a novel dataset in which scenes are populated by ShapeNet models. We find that after training ObSuRF on RGB-D views of training scenes, it is capable of not only recovering the 3D geometry of a scene depicted in a single input image, but also to segment it into objects, despite receiving no supervision in that regard.

READ FULL TEXT

page 1

page 4

page 8

research
12/03/2020

pixelNeRF: Neural Radiance Fields from One or Few Images

We propose pixelNeRF, a learning framework that predicts a continuous ne...
research
06/07/2022

ObPose: Leveraging Canonical Pose for Object-Centric Scene Inference in 3D

We present ObPose, an unsupervised object-centric generative model that ...
research
04/16/2018

IterGANs: Iterative GANs to Learn and Control 3D Object Transformation

We are interested in learning visual representations which allow for 3D ...
research
09/21/2017

SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes

This paper presents SceneCut, a novel approach to jointly discover previ...
research
04/16/2016

Automatic Segmentation of Dynamic Objects from an Image Pair

Automatic segmentation of objects from a single image is a challenging p...
research
03/21/2022

Generating Fast and Slow: Scene Decomposition via Reconstruction

We consider the problem of segmenting scenes into constituent entities, ...
research
04/27/2020

CoReNet: Coherent 3D scene reconstruction from a single RGB image

Advances in deep learning techniques have allowed recent work to reconst...

Please sign up or login with your details

Forgot password? Click here to reset