Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography

12/22/2022
by   Ilya Chugunov, et al.
0

Modern mobile burst photography pipelines capture and merge a short sequence of frames to recover an enhanced image, but often disregard the 3D nature of the scene they capture, treating pixel motion between images as a 2D aggregation problem. We show that in a "long-burst", forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. To this end, we devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion. Our plane plus depth model is trained end-to-end, and performs coarse-to-fine refinement by controlling which multi-resolution volume features the network has access to at what time during training. We validate the method experimentally, and demonstrate geometrically accurate depth reconstructions with no additional hardware or separate data pre-processing and pose-estimation steps.

READ FULL TEXT

page 6

page 7

page 8

page 13

page 15

page 16

page 18

page 19

research
11/26/2021

The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement

Modern smartphones can continuously stream multi-megapixel RGB images at...
research
02/11/2019

A Motion Free Approach to Dense Depth Estimation in Complex Dynamic Scene

Despite the recent success in per-frame monocular dense depth estimation...
research
04/29/2021

The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

Self-supervised monocular depth estimation networks are trained to predi...
research
08/04/2018

Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

We propose a CNN-based approach for multi-camera markerless motion captu...
research
07/08/2020

When Perspective Comes for Free: Improving Depth Prediction with Camera Pose Encoding

Monocular depth prediction is a highly underdetermined problem and recen...
research
04/07/2023

DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium

Self-supervised multi-frame depth estimation achieves high accuracy by c...
research
03/31/2023

Single Image Depth Prediction Made Better: A Multivariate Gaussian Take

Neural-network-based single image depth prediction (SIDP) is a challengi...

Please sign up or login with your details

Forgot password? Click here to reset