Self-Supervised Moving Vehicle Detection from Audio-Visual Cues

by   Jannik Zürn, et al.

Robust detection of moving vehicles is a critical task for any autonomously operating outdoor robot or self-driving vehicle. Most modern approaches for solving this task rely on training image-based detectors using large-scale vehicle detection datasets such as nuScenes or the Waymo Open Dataset. Providing manual annotations is an expensive and laborious exercise that does not scale well in practice. To tackle this problem, we propose a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. Our approach employs contrastive learning for localizing vehicles in images from corresponding pairs of images and recorded audio. In extensive experiments carried out with a real-world dataset, we demonstrate that our approach provides accurate detections of moving vehicles and does not require manual annotations. We furthermore show that our model can be used as a teacher to supervise an audio-only detection model. This student model is invariant to illumination changes and thus effectively bridges the domain gap inherent to models leveraging exclusively vision as the predominant modality.


page 1

page 3

page 4

page 6


Self-supervised Moving Vehicle Tracking with Stereo Sound

Humans are able to localize objects in the environment using both visual...

Self-Supervised Learning for Videos: A Survey

The remarkable success of deep learning in various domains relies on the...

Pi-PE: A Pipeline for Pulmonary Embolism Detection using Sparsely Annotated 3D CT Images

Pulmonary embolisms (PE) are known to be one of the leading causes for c...

Real-Time Idling Vehicles Detection Using Combined Audio-Visual Deep Learning

Combustion vehicle emissions contribute to poor air quality and release ...

Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning

Large-scale datasets are the cornerstone of self-supervised representati...

Please sign up or login with your details

Forgot password? Click here to reset