In-Place Scene Labelling and Understanding with Implicit Scene Representation

03/29/2021
by   Shuaifeng Zhi, et al.
0

Semantic labelling is highly correlated with geometry and radiance reconstruction, as scene entities with similar shape and appearance are more likely to come from similar classes. Recent implicit neural reconstruction techniques are appealing as they do not require prior training data, but the same fully self-supervised approach is not possible for semantics because labels are human-defined properties. We extend neural radiance fields (NeRF) to jointly encode semantics with appearance and geometry, so that complete and accurate 2D semantic labels can be achieved using a small amount of in-place annotations specific to the scene. The intrinsic multi-view consistency and smoothness of NeRF benefit semantics by enabling sparse labels to efficiently propagate. We show the benefit of this approach when labels are either sparse or very noisy in room-scale scenes. We demonstrate its advantageous properties in various interesting applications such as an efficient scene labelling tool, novel semantic view synthesis, label denoising, super-resolution, label interpolation and multi-view semantic label fusion in visual semantic mapping systems.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 6

page 7

page 8

page 11

page 12

page 13

page 14

11/29/2021

iLabel: Interactive Neural Scene Labelling

Joint representation of geometry, colour and semantics using a 3D neural...
02/24/2020

Comparing View-Based and Map-Based Semantic Labelling in Real-Time SLAM

Generally capable Spatial AI systems must build persistent scene represe...
05/05/2022

Neural 3D Scene Reconstruction with the Manhattan-world Assumption

This paper addresses the challenge of reconstructing 3D indoor scenes fr...
12/04/2017

3D Semantic Trajectory Reconstruction from 3D Pixel Continuum

This paper presents a method to reconstruct dense semantic trajectory st...
03/29/2022

Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Large-scale training data with high-quality annotations is critical for ...
03/15/2019

SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representation

Systems which incrementally create 3D semantic maps from image sequences...
03/15/2019

SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations

Systems which incrementally create 3D semantic maps from image sequences...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.