RelightableHands: Efficient Neural Relighting of Articulated Hand Models

by   Shun Iwase, et al.

We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage, allowing us to synthesize hands in arbitrary illuminations but with heavy compute. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time. To achieve generalization, we condition the student model with physics-inspired illumination features such as visibility, diffuse shading, and specular reflections computed on a coarse proxy geometry, maintaining a small computational overhead. Our key insight is that these features have strong correlation with subsequent global light transport effects, which proves sufficient as conditioning data for the neural relighting network. Moreover, in contrast to bottleneck illumination conditioning, these features are spatially aligned based on underlying geometry, leading to better generalization to unseen illuminations and poses. In our experiments, we demonstrate the efficacy of our illumination feature representations, outperforming baseline approaches. We also show that our approach can photorealistically relight two interacting hands at real-time speeds.


page 1

page 6

page 7

page 8

page 14

page 15

page 16

page 17


Feature Matters: A Stage-by-Stage Approach for Knowledge Transfer

Convolutional Neural Networks (CNNs) become deeper and deeper in recent ...

Illumination adaptive person reid based on teacher-student model and adversarial training

Most existing works in Person Re-identification (ReID) focus on settings...

PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

We present PhySG, an end-to-end inverse rendering pipeline that includes...

MEGANE: Morphable Eyeglass and Avatar Network

Eyeglasses play an important role in the perception of identity. Authent...

Towards Practical Capture of High-Fidelity Relightable Avatars

In this paper, we propose a novel framework, Tracking-free Relightable A...

Deep-learning the Latent Space of Light Transport

We suggest a method to directly deep-learn light transport, i. e., the m...

Light Efficient Flutter Shutter

Flutter shutter is a technique in which the exposure is chopped into seg...

Please sign up or login with your details

Forgot password? Click here to reset