Towards Panoptic 3D Parsing for Single Image in the Wild

11/04/2021
by   Sainan Liu, et al.
2

Performing single image holistic understanding and 3D reconstruction is a central task in computer vision. This paper presents an integrated system that performs holistic image segmentation, object detection, instance segmentation, depth estimation, and object instance 3D reconstruction for indoor and outdoor scenes from a single RGB image. We name our system panoptic 3D parsing in which panoptic segmentation ("stuff" segmentation and "things" detection/segmentation) with 3D reconstruction is performed. We design a stage-wise system where a complete set of annotations is absent. Additionally, we present an end-to-end pipeline trained on a synthetic dataset with a full set of annotations. We show results on both indoor (3D-FRONT) and outdoor (COCO and Cityscapes) scenes. Our proposed panoptic 3D parsing framework points to a promising direction in computer vision. It can be applied to various applications, including autonomous driving, mapping, robotics, design, computer graphics, robotics, human-computer interaction, and augmented reality.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 6

page 7

research
11/03/2021

Panoptic 3D Scene Reconstruction From a Single RGB Image

Understanding 3D scenes from a single image is fundamental to a wide var...
research
10/06/2020

Parallax Motion Effect Generation Through Instance Segmentation And Depth Estimation

Stereo vision is a growing topic in computer vision due to the innumerab...
research
06/07/2019

Deep Robust Single Image Depth Estimation Neural Network Using Scene Understanding

Single image depth estimation (SIDE) plays a crucial role in 3D computer...
research
07/16/2020

PerMO: Perceiving More at Once from a Single Image for Autonomous Driving

We present a novel approach to detect, segment, and reconstruct complete...
research
07/11/2018

Deep attention-based classification network for robust depth prediction

In this paper, we present our deep attention-based classification (DABC)...
research
09/24/2010

Modeling Instantaneous Changes In Natural Scenes

This project aims to create 3d model of the natural world and model chan...
research
08/07/2018

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image

We propose a computational framework to jointly parse a single RGB image...

Please sign up or login with your details

Forgot password? Click here to reset