DeepAI AI Chat
Log In Sign Up

SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representation

by   Shuaifeng Zhi, et al.
Imperial College London

Systems which incrementally create 3D semantic maps from image sequences must store and update representations of both geometry and semantic entities. However, while there has been much work on the correct formulation for geometrical estimation, state-of-the-art systems usually rely on simple semantic representations which store and update independent label estimates for each surface element (depth pixels, surfels, or voxels). Spatial correlation is discarded, and fused label maps are incoherent and noisy. We introduce a new compact and optimisable semantic representation by training a variational auto-encoder that is conditioned on a colour image. Using this learned latent space, we can tackle semantic label fusion by jointly optimising the low-dimenional codes associated with each of a set of overlapping images, producing consistent fused label maps which preserve spatial correlation. We also show how this approach can be used within a monocular keyframe based semantic mapping system where a similar code approach is used for geometry. The probabilistic formulation allows a flexible formulation where we can jointly estimate motion, geometry and semantics in a unified optimisation.


page 1

page 3

page 6

page 7

page 8


SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations

Systems which incrementally create 3D semantic maps from image sequences...

CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM

The representation of geometry in real-time 3D perception systems contin...

DeepFactors: Real-Time Probabilistic Dense Monocular SLAM

The ability to estimate rich geometry and camera motion from monocular i...

Object Structural Points Representation for Graph-based Semantic Monocular Localization and Mapping

Efficient object level representation for monocular semantic simultaneou...

In-Place Scene Labelling and Understanding with Implicit Scene Representation

Semantic labelling is highly correlated with geometry and radiance recon...

Towards Building the Semantic Map from a Monocular Camera with a Multi-task Network

In many robotic applications, especially for the autonomous driving, und...

From Pixels to Buildings: End-to-end Probabilistic Deep Networks for Large-scale Semantic Mapping

We introduce TopoNets, end-to-end probabilistic deep networks for modeli...