Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions

04/13/2022
by   Zipeng Ye, et al.
0

Recently, talking-face video generation has received considerable attention. So far most methods generate results with neutral expressions or expressions that are implicitly determined by neural networks in an uncontrollable way. In this paper, we propose a method to generate talking-face videos with continuously controllable expressions in real-time. Our method is based on an important observation: In contrast to facial geometry of moderate resolution, most expression information lies in textures. Then we make use of neural textures to generate high-quality talking face videos and design a novel neural network that can generate neural textures for image frames (which we called dynamic neural textures) based on the input expression and continuous intensity expression coding (CIEC). Our method uses 3DMM as a 3D model to sample the dynamic neural texture. The 3DMM does not cover the teeth area, so we propose a teeth submodule to complete the details in teeth. Results and an ablation study show the effectiveness of our method in generating high-quality talking-face videos with continuously controllable expressions. We also set up four baseline methods by combining existing representative methods and compare them with our method. Experimental results including a user study show that our method has the best performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

page 5

page 6

page 8

page 9

research
09/17/2022

Continuously Controllable Facial Expression Editing in Talking Face Videos

Recently audio-driven talking face video generation has attracted consid...
research
08/09/2018

Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation

The recent advances in deep learning have made it possible to generate p...
research
05/16/2021

3D to 4D Facial Expressions Generation Guided by Landmarks

While deep learning-based 3D face generation has made a progress recentl...
research
08/13/2020

Modeling Caricature Expressions by 3D Blendshape and Dynamic Texture

The problem of deforming an artist-drawn caricature according to a given...
research
10/01/2020

Dynamic Facial Asset and Rig Generation from a Single Scan

The creation of high-fidelity computer-generated (CG) characters used in...
research
11/19/2020

Assembling a Pipeline for 3D Face Interpolation

This paper describes a pipeline built with open source tools for interpo...
research
09/07/2023

Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance Fields using Geometry-Guided Text-to-Image Diffusion Model

Recent advances in diffusion models such as ControlNet have enabled geom...

Please sign up or login with your details

Forgot password? Click here to reset