DeLTra: Deep Light Transport for Projector-Camera Systems

03/06/2020
by   Bingyao Huang, et al.
9

In projector-camera systems, light transport models the propagation from projector emitted radiance to camera-captured irradiance. In this paper, we propose the first end-to-end trainable solution named Deep Light Transport (DeLTra) that estimates radiometrically uncalibrated projector-camera light transport. DeLTra is designed to have two modules: DepthToAtrribute and ShadingNet. DepthToAtrribute explicitly learns rays, depth and normal, and then estimates rough Phong illuminations. Afterwards, the CNN-based ShadingNet renders photorealistic camera-captured image using estimated shading attributes and rough Phong illuminations. A particular challenge addressed by DeLTra is occlusion, for which we exploit epipolar constraint and propose a novel differentiable direct light mask. Thus, it can be learned end-to-end along with the other DeLTra modules. Once trained, DeLTra can be applied simultaneously to three projector-camera tasks: image-based relighting, projector compensation and depth/normal reconstruction. In our experiments, DeLTra shows clear advantages over previous arts with promising quality and meanwhile being practically convenient.

READ FULL TEXT

page 8

page 12

page 13

page 14

research
08/17/2019

CompenNet++: End-to-end Full Projector Compensation

Full projector compensation aims to modify a projector input image such ...
research
12/11/2018

DeepV2D: Video to Depth with Differentiable Structure from Motion

We propose DeepV2D, an end-to-end differentiable deep learning architect...
research
07/30/2020

End-to-end Full Projector Compensation

Full projector compensation aims to modify a projector input image to co...
research
08/01/2021

Learned holographic light transport

Computer-Generated Holography (CGH) algorithms often fall short in match...
research
08/19/2021

PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility

Recent learning-based multi-view stereo (MVS) methods show excellent per...
research
11/25/2021

Facial Depth and Normal Estimation using Single Dual-Pixel Camera

Many mobile manufacturers recently have adopted Dual-Pixel (DP) sensors ...
research
03/21/2023

Automatic evaluation of herding behavior in towed fishing gear using end-to-end training of CNN and attention-based networks

This paper considers the automatic classification of herding behavior in...

Please sign up or login with your details

Forgot password? Click here to reset