PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

03/04/2019
by   Gaku Narita, et al.
0

We propose PanopticFusion, a novel online volumetric semantic mapping system at the level of stuff and things. In contrast to previous semantic mapping systems, PanopticFusion is able to densely predict class labels of a background region (stuff) and individually segment arbitrary foreground objects (things). In addition, our system has the capability to reconstruct a large-scale scene and extract a labeled mesh thanks to its use of a spatially hashed volumetric map representation. Our system first predicts pixel-wise panoptic labels (class labels for stuff regions and instance IDs for thing regions) for incoming RGB frames by fusing 2D semantic and instance segmentation outputs. The predicted panoptic labels are integrated into the volumetric map together with depth measurements while keeping the consistency of the instance IDs, which could vary frame to frame, by referring to the 3D map at that moment. In addition, we construct a fully connected conditional random field (CRF) model with respect to panoptic labels for map regularization. For online CRF inference, we propose a novel unary potential approximation and a map division strategy. We evaluated the performance of our system on the ScanNet (v2) dataset. PanopticFusion outperformed or compared with state-of-the-art offline 3D DNN methods in both semantic and instance segmentation benchmarks. Also, we demonstrate a promising augmented reality application using a 3D panoptic map generated by the proposed system.

READ FULL TEXT

page 1

page 6

page 7

research
03/01/2019

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

To autonomously navigate and plan interactions in real-world environment...
research
09/21/2021

Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency

For robotic interaction in an environment shared with multiple agents, a...
research
04/23/2018

MVTec D2S: Densely Segmented Supermarket Dataset

We introduce the Densely Segmented Supermarket (D2S) dataset, a novel be...
research
12/17/2018

3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans

We introduce 3D-SIS, a novel neural network architecture for 3D semantic...
research
06/20/2019

3D Instance Segmentation via Multi-task Metric Learning

We propose a novel method for instance label segmentation of dense 3D vo...
research
10/26/2018

Semantic Mapping with Simultaneous Object Detection and Localization

We present a filtering-based method for semantic mapping to simultaneous...
research
07/01/2015

Polarimetric Hierarchical Semantic Model and Scattering Mechanism Based PolSAR Image Classification

For polarimetric SAR (PolSAR) image classification, it is a challenge to...

Please sign up or login with your details

Forgot password? Click here to reset