DeepAI AI Chat
Log In Sign Up

Dense RGB-D semantic mapping with Pixel-Voxel neural network

by   Cheng Zhao, et al.

For intelligent robotics applications, extending 3D mapping to 3D semantic mapping enables robots to, not only localize themselves with respect to the scene's geometrical features but also simultaneously understand the higher level meaning of the scene contexts. Most previous methods focus on geometric 3D reconstruction and scene understanding independently notwithstanding the fact that joint estimation can boost the accuracy of the semantic mapping. In this paper, a dense RGB-D semantic mapping system with a Pixel-Voxel network is proposed, which can perform dense 3D mapping while simultaneously recognizing and semantically labelling each point in the 3D map. The proposed Pixel-Voxel network obtains global context information by using PixelNet to exploit the RGB image and meanwhile, preserves accurate local shape information by using VoxelNet to exploit the corresponding 3D point cloud. Unlike the existing architecture that fuses score maps from different models with equal weights, we proposed a Softmax weighted fusion stack that adaptively learns the varying contributions of PixelNet and VoxelNet, and fuses the score maps of the two models according to their respective confidence levels. The proposed Pixel-Voxel network achieves the state-of-the-art semantic segmentation performance on the SUN RGB-D benchmark dataset. The runtime of the proposed system can be boosted to 11-12Hz, enabling near to real-time performance using an i7 8-cores PC with Titan X GPU.


page 1

page 7

page 8


Semantic Dense Reconstruction with Consistent Scene Segments

In this paper, a method for dense semantic 3D scene reconstruction from ...

CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusion

Semantic scene completion (SSC) aims to complete a partial 3D scene and ...

DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks

3D scene understanding is important for robots to interact with the 3D w...

Atlas: End-to-End 3D Scene Reconstruction from Posed Images

We present an end-to-end 3D reconstruction method for a scene by directl...

Pixel Difference Convolutional Network for RGB-D Semantic Segmentation

RGB-D semantic segmentation can be advanced with convolutional neural ne...

3D Gated Recurrent Fusion for Semantic Scene Completion

This paper tackles the problem of data fusion in the semantic scene comp...

SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks

Ever more robust, accurate and detailed mapping using visual sensing has...