Facial Depth and Normal Estimation using Single Dual-Pixel Camera

11/25/2021
by   Minjun Kang, et al.
0

Many mobile manufacturers recently have adopted Dual-Pixel (DP) sensors in their flagship models for faster auto-focus and aesthetic image captures. Despite their advantages, research on their usage for 3D facial understanding has been limited due to the lack of datasets and algorithmic designs that exploit parallax in DP images. This is because the baseline of sub-aperture images is extremely narrow and parallax exists in the defocus blur region. In this paper, we introduce a DP-oriented Depth/Normal network that reconstructs the 3D facial geometry. For this purpose, we collect a DP facial data with more than 135K images for 101 persons captured with our multi-camera structured light systems. It contains the corresponding ground-truth 3D models including depth map and surface normal in metric scale. Our dataset allows the proposed matching network to be generalized for 3D facial depth/normal estimation. The proposed network consists of two novel modules: Adaptive Sampling Module and Adaptive Normal Module, which are specialized in handling the defocus blur in DP images. Finally, the proposed method achieves state-of-the-art performances over recent DP-based depth/normal estimation methods. We also demonstrate the applicability of the estimated depth/normal to face spoofing and relighting.

READ FULL TEXT

page 1

page 2

page 3

page 4

page 6

page 7

page 8

research
05/01/2020

Defocus Deblurring Using Dual-Pixel Data

Defocus blur arises in images that are captured with a shallow depth of ...
research
12/01/2020

Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration

The dual-pixel (DP) hardware works by splitting each pixel in half and c...
research
07/19/2023

LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network

Recovering sharp images from dual-pixel (DP) pairs with disparity-depend...
research
12/06/2020

Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data

Recent work has shown impressive results on data-driven defocus deblurri...
research
03/24/2016

Fine-scale Surface Normal Estimation using a Single NIR Image

We present surface normal estimation using a single near infrared (NIR) ...
research
08/11/2021

Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning

Many camera sensors use a dual-pixel (DP) design that operates as a rudi...
research
03/06/2020

DeLTra: Deep Light Transport for Projector-Camera Systems

In projector-camera systems, light transport models the propagation from...

Please sign up or login with your details

Forgot password? Click here to reset