SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations

03/15/2019
by   Shuaifeng Zhi, et al.
12

Systems which incrementally create 3D semantic maps from image sequences must store and update representations of both geometry and semantic entities. However, while there has been much work on the correct formulation for geometrical estimation, state-of-the-art systems usually rely on simple semantic representations which store and update independent label estimates for each surface element (depth pixels, surfels, or voxels). Spatial correlation is discarded, and fused label maps are incoherent and noisy. We introduce a new compact and optimisable semantic representation by training a variational auto-encoder that is conditioned on a colour image. Using this learned latent space, we can tackle semantic label fusion by jointly optimising the low-dimenional codes associated with each of a set of overlapping images, producing consistent fused label maps which preserve spatial correlation. We also show how this approach can be used within a monocular keyframe based semantic mapping system where a similar code approach is used for geometry. The probabilistic formulation allows a flexible formulation where we can jointly estimate motion, geometry and semantics in a unified optimisation.

READ FULL TEXT

page 1

page 3

page 6

page 7

page 8

research
03/15/2019

SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representation

Systems which incrementally create 3D semantic maps from image sequences...
research
04/03/2018

CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM

The representation of geometry in real-time 3D perception systems contin...
research
02/11/2021

Learning Depth via Leveraging Semantics: Self-supervised Monocular Depth Estimation with Both Implicit and Explicit Semantic Guidance

Self-supervised depth estimation has made a great success in learning de...
research
06/21/2022

Object Structural Points Representation for Graph-based Semantic Monocular Localization and Mapping

Efficient object level representation for monocular semantic simultaneou...
research
08/19/2023

Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing

The neural rendering of humans is a topic of great research significance...
research
07/27/2022

Towards the Probabilistic Fusion of Learned Priors into Standard Pipelines for 3D Reconstruction

The best way to combine the results of deep learning with standard 3D re...
research
12/18/2020

CodeVIO: Visual-Inertial Odometry with Learned Optimizable Dense Depth

In this work, we present a lightweight, tightly-coupled deep depth netwo...

Please sign up or login with your details

Forgot password? Click here to reset