Neural Animation and Reenactment of Human Actor Videos

09/11/2018
by   Lingjie Liu, et al.
4

We propose a method for generating (near) video-realistic animations of real humans under user control. In contrast to conventional human character rendering, we do not require the availability of a production-quality photo-realistic 3D model of the human, but instead rely on a video sequence in conjunction with a (medium-quality) controllable 3D template model of the person. With that, our approach significantly reduces production cost compared to conventional rendering approaches based on production-quality 3D models, and can also be used to realistically edit existing videos. Technically, this is achieved by training a neural network that translates simple synthetic images of a human character into realistic imagery. For training our networks, we first track the 3D motion of the person in the video using the template model, and subsequently generate a synthetically rendered version of the video. These images are then used to train a conditional generative adversarial network that translates synthetic images of the 3D model into realistic imagery of the human. We evaluate our method for the reenactment of another person that is tracked in order to obtain the motion data, and show video results generated from artist-designed skeleton motion. Our results outperform the state-of-the-art in learning-based human image synthesis. Project page: http://gvv.mpi-inf.mpg.de/projects/wxu/HumanReenactment/

READ FULL TEXT

page 1

page 4

page 7

page 8

page 9

page 10

page 11

research
12/11/2019

Neural Voice Puppetry: Audio-driven Facial Reenactment

We present Neural Voice Puppetry, a novel approach for audio-driven faci...
research
11/19/2020

Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

To be truly understandable and accepted by Deaf communities, an automati...
research
03/22/2022

QS-Craft: Learning to Quantize, Scrabble and Craft for Conditional Human Motion Animation

This paper studies the task of conditional Human Motion Animation (cHMA)...
research
07/17/2020

Personalized Speech2Video with 3D Skeleton Regularization and Expressive Body Poses

In this paper, we propose a novel approach to convert given speech audio...
research
03/30/2019

Dance Dance Generation: Motion Transfer for Internet Videos

This work presents computational methods for transferring body movements...
research
11/14/2018

Creatures great and SMAL: Recovering the shape and motion of animals from video

We present a system to recover the 3D shape and motion of a wide variety...
research
06/02/2015

What Makes Kevin Spacey Look Like Kevin Spacey

We reconstruct a controllable model of a person from a large photo colle...

Please sign up or login with your details

Forgot password? Click here to reset