FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

10/06/2022
by   Andreas Meuleman, et al.
0

High-accuracy per-pixel depth is vital for computational photography, so smartphones now have multimodal camera systems with time-of-flight (ToF) depth sensors and multiple color cameras. However, producing accurate high-resolution depth is still challenging due to the low resolution and limited active illumination power of ToF sensors. Fusing RGB stereo and ToF information is a promising direction to overcome these issues, but a key problem remains: to provide high-quality 2D RGB images, the main color sensor's lens is optically stabilized, resulting in an unknown pose for the floating lens that breaks the geometric relationships between the multimodal image sensors. Leveraging ToF depth estimates and a wide-angle RGB camera, we design an automatic calibration technique based on dense 2D/3D matching that can estimate camera extrinsic, intrinsic, and distortion parameters of a stabilized main RGB sensor from a single snapshot. This lets us fuse stereo and ToF cues via a correlation volume. For fusion, we apply deep learning via a real-world training dataset with depth supervision estimated by a neural reconstruction method. For evaluation, we acquire a test dataset using a commercial high-power depth camera and show that our approach achieves higher accuracy than existing baselines.

READ FULL TEXT

page 2

page 5

page 10

page 11

page 13

research
07/30/2021

High-Resolution Depth Maps Based on TOF-Stereo Fusion

The combination of range sensors with color cameras can be very useful f...
research
01/09/2019

Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera

Depth sensing is crucial for 3D reconstruction and scene understanding. ...
research
09/27/2022

Simultaneous Acquisition of High Quality RGB Image and Polarization Information using a Sparse Polarization Sensor

This paper proposes a novel polarization sensor structure and network ar...
research
09/17/2019

Deep End-to-End Alignment and Refinement for Time-of-Flight RGB-D Module

Recently, it is increasingly popular to equip mobile RGB cameras with Ti...
research
01/20/2017

Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras

Color-depth cameras (RGB-D cameras) have become the primary sensors in m...
research
11/26/2021

The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement

Modern smartphones can continuously stream multi-megapixel RGB images at...
research
05/24/2020

Learning Camera Miscalibration Detection

Self-diagnosis and self-repair are some of the key challenges in deployi...

Please sign up or login with your details

Forgot password? Click here to reset