ThermRad: A Multi-modal Dataset for Robust 3D Object Detection under Challenging Conditions

by   Qiao Yan, et al.

Robust 3D object detection in extreme weather and illumination conditions is a challenging task. While radars and thermal cameras are known for their resilience to these conditions, few studies have been conducted on radar-thermal fusion due to the lack of corresponding datasets. To address this gap, we first present a new multi-modal dataset called ThermRad, which includes a 3D LiDAR, a 4D radar, an RGB camera and a thermal camera. This dataset is unique because it includes data from all four sensors in extreme weather conditions, providing a valuable resource for future research in this area. To validate the robustness of 4D radars and thermal cameras for 3D object detection in challenging weather conditions, we propose a new multi-modal fusion method called RTDF-RCNN, which leverages the complementary strengths of 4D radars and thermal cameras to boost object detection performance. To further prove the effectiveness of our proposed framework, we re-implement state-of-the-art (SOTA) 3D detectors on our dataset as benchmarks for evaluation. Our method achieves significant enhancements in detecting cars, pedestrians, and cyclists, with improvements of over 7.98 respectively, while achieving comparable results to LiDAR-based approaches. Our contributions in both the ThermRad dataset and the new multi-modal fusion method provide a new approach to robust 3D object detection in adverse weather and illumination conditions. The ThermRad dataset will be released.


page 1

page 4

page 5

page 8


DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars

We propose DeepFusion, a modular multi-modal architecture to fuse lidars...

Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery

Can we improve detection in the thermal domain by borrowing features fro...

Robust Human Detection under Visual Degradation via Thermal and mmWave Radar Fusion

The majority of human detection methods rely on the sensor using visible...

WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness Search and Rescue

Sensor-equipped unoccupied aerial vehicles (UAVs) have the potential to ...

Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving

Multi-modal fusion has shown initial promising results for object detect...

LiDAR-guided object search and detection in Subterranean Environments

Detecting objects of interest, such as human survivors, safety equipment...

Fail-Safe Human Detection for Drones Using a Multi-Modal Curriculum Learning Approach

Drones are currently being explored for safety-critical applications whe...

Please sign up or login with your details

Forgot password? Click here to reset