Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers

05/12/2022
by   Robin Kips, et al.
8

Augmented reality applications have rapidly spread across online platforms, allowing consumers to virtually try-on a variety of products, such as makeup, hair dying, or shoes. However, parametrizing a renderer to synthesize realistic images of a given product remains a challenging task that requires expert knowledge. While recent work has introduced neural rendering methods for virtual try-on from example images, current approaches are based on large generative models that cannot be used in real-time on mobile devices. This calls for a hybrid method that combines the advantages of computer graphics and neural rendering approaches. In this paper we propose a novel framework based on deep learning to build a real-time inverse graphics encoder that learns to map a single example image into the parameter space of a given augmented reality rendering engine. Our method leverages self-supervised learning and does not require labeled training data which makes it extendable to many virtual try-on applications. Furthermore, most augmented reality renderers are not differentiable in practice due to algorithmic choices or implementation constraints to reach real-time on portable devices. To relax the need for a graphics-based differentiable renderer in inverse graphics problems, we introduce a trainable imitator module. Our imitator is a generative network that learns to accurately reproduce the behavior of a given non-differentiable renderer. We propose a novel rendering sensitivity loss to train the imitator, which ensures that the network learns an accurate and continuous representation for each rendering parameter. Our framework enables novel applications where consumers can virtually try-on a novel unknown product from an inspirational reference image on social media. It can also be used by graphics artists to automatically create realistic rendering from a reference product image.

READ FULL TEXT

page 2

page 3

page 6

page 8

page 9

research
05/12/2021

Deep Graphics Encoder for Real-Time Video Makeup Synthesis from Example

While makeup virtual-try-on is now widespread, parametrizing a computer ...
research
02/08/2022

Hair Color Digitization through Imaging and Deep Inverse Graphics

Hair appearance is a complex phenomenon due to hair geometry and how the...
research
12/19/2017

Real-time deep hair matting on mobile devices

Augmented reality is an emerging technology in many application domains....
research
05/02/2023

DreamPaint: Few-Shot Inpainting of E-Commerce Items for Virtual Try-On without 3D Modeling

We introduce DreamPaint, a framework to intelligently inpaint any e-comm...
research
02/02/2023

Towards Attention-aware Rendering for Virtual and Augmented Reality

Foveated graphics is a promising approach to solving the bandwidth chall...
research
02/28/2020

Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data

Recent work has shown the ability to learn generative models for 3D shap...
research
01/04/2021

Single-shot fringe projection profilometry based on Deep Learning and Computer Graphics

Multiple works have applied deep learning to fringe projection profilome...

Please sign up or login with your details

Forgot password? Click here to reset