Log In Sign Up

CroMo: Cross-Modal Learning for Monocular Depth Estimation

by   Yannick Verdie, et al.

Learning-based depth estimation has witnessed recent progress in multiple directions; from self-supervision using monocular video to supervised methods offering highest accuracy. Complementary to supervision, further boosts to performance and robustness are gained by combining information from multiple signals. In this paper we systematically investigate key trade-offs associated with sensor and modality design choices as well as related model training strategies. Our study leads us to a new method, capable of connecting modality-specific advantages from polarisation, Time-of-Flight and structured-light inputs. We propose a novel pipeline capable of estimating depth from monocular polarisation for which we evaluate various training signals. The inversion of differentiable analytic models thereby connects scene geometry with polarisation and ToF signals and enables self-supervised and cross-modal learning. In the absence of existing multimodal datasets, we examine our approach with a custom-made multi-modal camera rig and collect CroMo; the first dataset to consist of synchronized stereo polarisation, indirect ToF and structured-light depth, captured at video rates. Extensive experiments on challenging video scenes confirm both qualitative and quantitative pipeline advantages where we are able to outperform competitive monocular depth estimation method.


page 1

page 4

page 7

page 8


Self-Attention Dense Depth Estimation Network for Unrectified Video Sequences

The dense depth estimation of a 3D scene has numerous applications, main...

MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation

Depth-aware video panoptic segmentation tackles the inverse projection p...

R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes

While self-supervised monocular depth estimation in driving scenarios ha...

P2D: a self-supervised method for depth estimation from polarimetry

Monocular depth estimation is a recurring subject in the field of comput...

Beyond Image to Depth: Improving Depth Prediction using Echoes

We address the problem of estimating depth with multi modal audio visual...

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

The success of monocular depth estimation relies on large and diverse tr...