S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds

12/16/2020
by   Ran Cheng, et al.
1

With the increasing reliance of self-driving and similar robotic systems on robust 3D vision, the processing of LiDAR scans with deep convolutional neural networks has become a trend in academia and industry alike. Prior attempts on the challenging Semantic Scene Completion task - which entails the inference of dense 3D structure and associated semantic labels from "sparse" representations - have been, to a degree, successful in small indoor scenes when provided with dense point clouds or dense depth maps often fused with semantic segmentation maps from RGB images. However, the performance of these systems drop drastically when applied to large outdoor scenes characterized by dynamic and exponentially sparser conditions. Likewise, processing of the entire sparse volume becomes infeasible due to memory limitations and workarounds introduce computational inefficiency as practitioners are forced to divide the overall volume into multiple equal segments and infer on each individually, rendering real-time performance impossible. In this work, we formulate a method that subsumes the sparsity of large-scale environments and present S3CNet, a sparse convolution based neural network that predicts the semantically completed scene from a single, unified LiDAR point cloud. We show that our proposed method outperforms all counterparts on the 3D task, achieving state-of-the art results on the SemanticKITTI benchmark. Furthermore, we propose a 2D variant of S3CNet with a multi-view fusion strategy to complement our 3D network, providing robustness to occlusions and extreme sparsity in distant regions. We conduct experiments for the 2D semantic scene completion task and compare the results of our sparse 2D network against several leading LiDAR segmentation models adapted for bird's eye view segmentation on two open-source datasets.

READ FULL TEXT

page 2

page 4

page 5

page 6

page 7

page 12

page 13

08/02/2018

Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation

Convolutional neural networks are designed for dense data, but vision da...
04/05/2020

MNEW: Multi-domain Neighborhood Embedding and Weighting for Sparse Point Clouds Segmentation

Point clouds have been widely adopted in 3D semantic scene understanding...
11/18/2020

Semantic Scene Completion using Local Deep Implicit Functions on LiDAR Data

Semantic scene completion is the task of jointly estimating 3D geometry ...
11/22/2021

Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping

Paris-CARLA-3D is a dataset of several dense colored point clouds of out...
03/14/2022

MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments

This work addresses a gap in semantic scene completion (SSC) data by cre...
05/29/2019

A survey of Object Classification and Detection based on 2D/3D data

Recently, by using deep neural network based algorithms, object classifi...
08/24/2020

LMSCNet: Lightweight Multiscale 3D Semantic Completion

We introduce a new approach for multiscale 3D semantic scene completion ...