Consistent Direct Time-of-Flight Video Depth Super-Resolution

11/16/2022
by   Zhanghao Sun, et al.
0

Direct time-of-flight (dToF) sensors are promising for next-generation on-device 3D sensing. However, to achieve the sufficient signal-to-noise-ratio (SNR) in a compact module, the dToF data has limited spatial resolution (e.g.,  20x30 for iPhone dToF), and it requires a super-resolution step before being passed to downstream tasks. In this paper, we solve this super-resolution problem by fusing the low-resolution dToF data with the corresponding high-resolution RGB guidance. Unlike the conventional RGB-guided depth enhancement approaches which perform the fusion in a per-frame manner, we propose the first multi-frame fusion scheme to mitigate the spatial ambiguity resulting from the low-resolution dToF imaging. In addition, dToF sensors provide unique depth histogram information for each local patch, and we incorporate this dToF-specific feature in our network design to further alleviate spatial ambiguity. To evaluate our models on complex dynamic indoor environments and to provide a large-scale dToF sensor dataset, we introduce DyDToF, the first synthetic RGB-dToF video dataset that features dynamic objects and a realistic dToF simulator following the physical imaging process. We believe the methods and dataset are beneficial to a broad community as dToF depth sensing is becoming mainstream on mobile devices.

READ FULL TEXT

page 1

page 5

page 6

page 7

page 8

research
08/01/2017

Depth Super-Resolution Meets Uncalibrated Photometric Stereo

A novel depth super-resolution approach for RGB-D sensors is presented. ...
research
11/20/2020

Robust super-resolution depth imaging via a multi-feature fusion deep network

Three-dimensional imaging plays an important role in imaging application...
research
12/21/2018

Multi-Frame Super-Resolution Reconstruction with Applications to Medical Imaging

The optical resolution of a digital camera is one of its most crucial pa...
research
10/19/2022

Video super-resolution for single-photon LIDAR

3D Time-of-Flight (ToF) image sensors are used widely in applications su...
research
03/13/2019

Reactive Sensing and Multiplicative Frame Super-resolution

The problem is to evaluate the behavior of an object when primary source...
research
09/27/2022

DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image

Light-weight time-of-flight (ToF) depth sensors are small, cheap, low-en...
research
08/23/2023

StofNet: Super-resolution Time of Flight Network

Time of Flight (ToF) is a prevalent depth sensing technology in the fiel...

Please sign up or login with your details

Forgot password? Click here to reset