Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging

by   Ilya Chugunov, et al.

We introduce Mask-ToF, a method to reduce flying pixels (FP) in time-of-flight (ToF) depth captures. FPs are pervasive artifacts which occur around depth edges, where light paths from both an object and its background are integrated over the aperture. This light mixes at a sensor pixel to produce erroneous depth estimates, which can adversely affect downstream 3D vision tasks. Mask-ToF starts at the source of these FPs, learning a microlens-level occlusion mask which effectively creates a custom-shaped sub-aperture for each sensor pixel. This modulates the selection of foreground and background light mixtures on a per-pixel basis and thereby encodes scene geometric information directly into the ToF measurements. We develop a differentiable ToF simulator to jointly train a convolutional neural network to decode this information and produce high-fidelity, low-FP depth reconstructions. We test the effectiveness of Mask-ToF on a simulated light field dataset and validate the method with an experimental prototype. To this end, we manufacture the learned amplitude mask and design an optical relay system to virtually place it on a high-resolution ToF sensor. We find that Mask-ToF generalizes well to real data without retraining, cutting FP counts in half.



page 1

page 5

page 6

page 7

page 8


Toward Depth Estimation Using Mask-Based Lensless Cameras

Recently, coded masks have been used to demonstrate a thin form-factor l...

Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems

Compressive imaging using coded apertures (CA) is a powerful technique t...

A Simple Framework for 3D Lensless Imaging with Programmable Masks

Lensless cameras provide a framework to build thin imaging systems by re...

FlatCam: Thin, Bare-Sensor Cameras using Coded Aperture and Computation

FlatCam is a thin form-factor lensless camera that consists of a coded m...

All-photon Polarimetric Time-of-Flight Imaging

Time-of-flight (ToF) sensors provide an imaging modality fueling diverse...

Causes and Corrections for Bimodal Multipath Scanning with Structured Light

Structured light illumination is an active 3-D scanning technique based ...

Learning Pixel-Adaptive Weights for Portrait Photo Retouching

Portrait photo retouching is a photo retouching task that emphasizes hum...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.