Panoptic 3D Scene Reconstruction From a Single RGB Image

11/03/2021
by   Manuel Dahnert, et al.
0

Understanding 3D scenes from a single image is fundamental to a wide variety of tasks, such as for robotics, motion planning, or augmented reality. Existing works in 3D perception from a single RGB image tend to focus on geometric reconstruction only, or geometric reconstruction with semantic segmentation or instance segmentation. Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations. We thus propose a new approach for holistic 3D scene understanding from a single RGB image which learns to lift and propagate 2D features from an input image to a 3D volumetric scene representation. We demonstrate that this holistic view of joint scene reconstruction, semantic, and instance segmentation is beneficial over treating the tasks independently, thus outperforming alternative approaches.

READ FULL TEXT

page 2

page 4

page 8

page 9

page 13

page 15

research
11/04/2021

Towards Panoptic 3D Parsing for Single Image in the Wild

Performing single image holistic understanding and 3D reconstruction is ...
research
01/21/2020

Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation

We propose a geometric superstructure for unified real-time processing o...
research
07/20/2020

Can we cover navigational perception needs of the visually impaired by panoptic segmentation?

Navigational perception for visually impaired people has been substantia...
research
07/23/2019

U4D: Unsupervised 4D Dynamic Scene Understanding

We introduce the first approach to solve the challenging problem of unsu...
research
02/07/2019

Single Network Panoptic Segmentation for Street Scene Understanding

In this work, we propose a single deep neural network for panoptic segme...
research
04/14/2020

Footprints and Free Space from a Single Color Image

Understanding the shape of a scene from a single color image is a formid...
research
10/12/2022

Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image

We present a deep reinforcement learning method of progressive view inpa...

Please sign up or login with your details

Forgot password? Click here to reset