Neural Human Video Rendering: Joint Learning of Dynamic Textures and Rendering-to-Video Translation

01/14/2020
by   Lingjie Liu, et al.
1

Synthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.

READ FULL TEXT

page 2

page 4

page 5

page 7

page 8

page 9

page 10

research
01/14/2020

Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

Synthesizing realistic videos of humans using neural networks has been a...
research
06/27/2021

Robust Pose Transfer with Dynamic Details using Neural Video Rendering

Pose transfer of human videos aims to generate a high fidelity video of ...
research
04/28/2019

Deferred Neural Rendering: Image Synthesis using Neural Textures

The modern computer graphics pipeline can synthesize images at remarkabl...
research
03/27/2023

Generalizable Neural Voxels for Fast Human Radiance Fields

Rendering moving human bodies at free viewpoints only from a monocular v...
research
11/10/2021

Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis

Synthesizing dynamic appearances of humans in motion plays a central rol...
research
12/23/2020

Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild

Given an "in-the-wild" video of a person, we reconstruct an animatable m...
research
12/07/2020

Pose-Guided Human Animation from a Single Image in the Wild

We present a new pose transfer method for synthesizing a human animation...

Please sign up or login with your details

Forgot password? Click here to reset