Audio Input Generates Continuous Frames to Synthesize Facial Video Using Generative Adiversarial Networks

07/18/2022
by   Hanhaodi Zhang, et al.
0

This paper presents a simple method for speech videos generation based on audio: given a piece of audio, we can generate a video of the target face speaking this audio. We propose Generative Adversarial Networks (GAN) with cut speech audio input as condition and use Convolutional Gate Recurrent Unit (GRU) in generator and discriminator. Our model is trained by exploiting the short audio and the frames in this duration. For training, we cut the audio and extract the face in the corresponding frames. We designed a simple encoder and compare the generated frames using GAN with and without GRU. We use GRU for temporally coherent frames and the results show that short audio can produce relatively realistic output results.

READ FULL TEXT
research
05/08/2017

You said that?

We present a method for generating a video of a talking face. The method...
research
03/20/2018

Speech-Driven Facial Reenactment Using Conditional Generative Adversarial Networks

We present a novel approach to generating photo-realistic images of a fa...
research
04/06/2021

Strumming to the Beat: Audio-Conditioned Contrastive Video Textures

We introduce a non-parametric approach for infinite video texture synthe...
research
03/25/2019

Wav2Pix: Speech-conditioned Face Generation using Generative Adversarial Networks

Speech is a rich biometric signal that contains information about the id...
research
05/04/2021

VQCPC-GAN: Variable-length Adversarial Audio Synthesis using Vector-Quantized Contrastive Predictive Coding

Influenced by the field of Computer Vision, Generative Adversarial Netwo...
research
06/11/2021

Catch-A-Waveform: Learning to Generate Audio from a Single Short Example

Models for audio generation are typically trained on hours of recordings...
research
12/14/2020

Robust One Shot Audio to Video Generation

Audio to Video generation is an interesting problem that has numerous ap...

Please sign up or login with your details

Forgot password? Click here to reset