MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

03/15/2023
by   Jiale Li, et al.
0

LiDAR and camera are two modalities available for 3D semantic segmentation in autonomous driving. The popular LiDAR-only methods severely suffer from inferior segmentation on small and distant objects due to insufficient laser points, while the robust multi-modal solution is under-explored, where we investigate three crucial inherent difficulties: modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. We propose a multi-modal 3D semantic segmentation model (MSeg3D) with joint intra-modal feature extraction and inter-modal feature fusion to mitigate the modality heterogeneity. The multi-modal fusion in MSeg3D consists of geometry-based feature fusion GF-Phase, cross-modal feature completion, and semantic-based feature fusion SF-Phase on all visible points. The multi-modal data augmentation is reinvigorated by applying asymmetric transformations on LiDAR point cloud and multi-camera images individually, which benefits the model training with diversified augmentation transformations. MSeg3D achieves state-of-the-art results on nuScenes, Waymo, and SemanticKITTI datasets. Under the malfunctioning multi-camera input and the multi-frame point clouds input, MSeg3D still shows robustness and improves the LiDAR-only baseline. Our code is publicly available at <https://github.com/jialeli1/lidarseg3d>.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/10/2022

2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds

As camera and LiDAR sensors capture complementary information used in au...
research
11/22/2022

LiCamGait: Gait Recognition in the Wild by Using LiDAR and Camera Multi-modal Visual Sensors

LiDAR can capture accurate depth information in large-scale scenarios wi...
research
04/06/2023

Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentation

3D semantic segmentation is a critical task in many real-world applicati...
research
08/24/2023

SkipcrossNets: Adaptive Skip-cross Fusion for Road Detection

Multi-modal fusion is increasingly being used for autonomous driving tas...
research
11/12/2021

Multimodal Virtual Point 3D Detection

Lidar-based sensing drives current autonomous vehicles. Despite rapid pr...
research
01/31/2020

Universal Semantic Segmentation for Fisheye Urban Driving Images

Semantic segmentation is a critical method in the field of autonomous dr...
research
08/23/2023

SUMMIT: Source-Free Adaptation of Uni-Modal Models to Multi-Modal Targets

Scene understanding using multi-modal data is necessary in many applicat...

Please sign up or login with your details

Forgot password? Click here to reset