Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation

08/09/2018
by   Lijie Fan, et al.
6

The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip. In this paper, for the sake of both furthering this exploration and our own interest in a realistic application, we study image-to-video translation and particularly focus on the videos of facial expressions. This problem challenges the deep neural networks by another temporal dimension comparing to the image-to-image translation. Moreover, its single input image fails most existing video generation methods that rely on recurrent models. We propose a user-controllable approach so as to generate video clips of various lengths from a single face image. The lengths and types of the expressions are controlled by users. To this end, we design a novel neural network architecture that can incorporate the user input into its skip connections and propose several improvements to the adversarial training method for the neural network. Experiments and user studies verify the effectiveness of our approach. Especially, we would like to highlight that even for the face images in the wild (downloaded from the Web and the authors' own photos), our model can generate high-quality facial expression videos of which about 50% are labeled as real by Amazon Mechanical Turk workers.

READ FULL TEXT

page 6

page 7

page 8

research
04/13/2022

Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions

Recently, talking-face video generation has received considerable attent...
research
07/26/2018

Learning to Forecast and Refine Residual Motion for Image-to-Video Generation

We consider the problem of image-to-video translation, where an input im...
research
11/25/2022

Dynamic Neural Portraits

We present Dynamic Neural Portraits, a novel approach to the problem of ...
research
07/19/2020

Learning to Generate Customized Dynamic 3D Facial Expressions

Recent advances in deep learning have significantly pushed the state-of-...
research
09/07/2023

Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance Fields using Geometry-Guided Text-to-Image Diffusion Model

Recent advances in diffusion models such as ControlNet have enabled geom...
research
09/17/2022

Continuously Controllable Facial Expression Editing in Talking Face Videos

Recently audio-driven talking face video generation has attracted consid...
research
12/01/2021

Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos

In this paper, we introduce a novel deep learning method for photo-reali...

Please sign up or login with your details

Forgot password? Click here to reset