HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images

by   Johan Vertens, et al.

The majority of learning-based semantic segmentation methods are optimized for daytime scenarios and favorable lighting conditions. Real-world driving scenarios, however, entail adverse environmental conditions such as nighttime illumination or glare which remain a challenge for existing approaches. In this work, we propose a multimodal semantic segmentation model that can be applied during daytime and nighttime. To this end, besides RGB images, we leverage thermal images, making our network significantly more robust. We avoid the expensive annotation of nighttime images by leveraging an existing daytime RGB-dataset and propose a teacher-student training approach that transfers the dataset's knowledge to the nighttime domain. We further employ a domain adaptation method to align the learned feature spaces across the domains and propose a novel two-stage training scheme. Furthermore, due to a lack of thermal data for autonomous driving, we present a new dataset comprising over 20,000 time-synchronized and aligned RGB-thermal image pairs. In this context, we also present a novel target-less calibration method that allows for automatic robust extrinsic and intrinsic thermal camera calibration. Among others, we employ our new dataset to show state-of-the-art results for nighttime semantic segmentation.


page 1

page 4

page 6

page 7


PST900: RGB-Thermal Calibration, Dataset and Segmentation Network

In this work we propose long wave infrared (LWIR) imagery as a viable su...

Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation

The ability to scene understanding in adverse visual conditions, e.g., n...

DooDLeNet: Double DeepLab Enhanced Feature Fusion for Thermal-color Semantic Segmentation

In this paper we present a new approach for feature fusion between RGB a...

Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation

Despite much research progress in image semantic segmentation, it remain...

Semantic Segmentation for Thermal Images: A Comparative Survey

Semantic segmentation is a challenging task since it requires excessivel...

Unsupervised RGB-to-Thermal Domain Adaptation via Multi-Domain Attention Network

This work presents a new method for unsupervised thermal image classific...

BTSeg: Barlow Twins Regularization for Domain Adaptation in Semantic Segmentation

Semantic image segmentation is a critical component in many computer vis...

Please sign up or login with your details

Forgot password? Click here to reset