S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds

by   Ran Cheng, et al.

With the increasing reliance of self-driving and similar robotic systems on robust 3D vision, the processing of LiDAR scans with deep convolutional neural networks has become a trend in academia and industry alike. Prior attempts on the challenging Semantic Scene Completion task - which entails the inference of dense 3D structure and associated semantic labels from "sparse" representations - have been, to a degree, successful in small indoor scenes when provided with dense point clouds or dense depth maps often fused with semantic segmentation maps from RGB images. However, the performance of these systems drop drastically when applied to large outdoor scenes characterized by dynamic and exponentially sparser conditions. Likewise, processing of the entire sparse volume becomes infeasible due to memory limitations and workarounds introduce computational inefficiency as practitioners are forced to divide the overall volume into multiple equal segments and infer on each individually, rendering real-time performance impossible. In this work, we formulate a method that subsumes the sparsity of large-scale environments and present S3CNet, a sparse convolution based neural network that predicts the semantically completed scene from a single, unified LiDAR point cloud. We show that our proposed method outperforms all counterparts on the 3D task, achieving state-of-the art results on the SemanticKITTI benchmark. Furthermore, we propose a 2D variant of S3CNet with a multi-view fusion strategy to complement our 3D network, providing robustness to occlusions and extreme sparsity in distant regions. We conduct experiments for the 2D semantic scene completion task and compare the results of our sparse 2D network against several leading LiDAR segmentation models adapted for bird's eye view segmentation on two open-source datasets.


page 2

page 4

page 5

page 6

page 7

page 12

page 13


Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation

Convolutional neural networks are designed for dense data, but vision da...

MNEW: Multi-domain Neighborhood Embedding and Weighting for Sparse Point Clouds Segmentation

Point clouds have been widely adopted in 3D semantic scene understanding...

Semantic Scene Completion using Local Deep Implicit Functions on LiDAR Data

Semantic scene completion is the task of jointly estimating 3D geometry ...

Semantic Segmentation-assisted Scene Completion for LiDAR Point Clouds

Outdoor scene completion is a challenging issue in 3D scene understandin...

Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping

Paris-CARLA-3D is a dataset of several dense colored point clouds of out...

MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments

This work addresses a gap in semantic scene completion (SSC) data by cre...

A survey of Object Classification and Detection based on 2D/3D data

Recently, by using deep neural network based algorithms, object classifi...

Please sign up or login with your details

Forgot password? Click here to reset