Speech-driven facial animation using polynomial fusion of features

12/12/2019
by   Triantafyllos Kefalas, et al.
15

Speech-driven facial animation involves using a speech signal to generate realistic videos of talking faces. Recent deep learning approaches to facial synthesis rely on extracting low-dimensional representations and concatenating them, followed by a decoding step of the concatenated vector. This accounts for only first-order interactions of the features and ignores higher-order interactions. In this paper we propose a polynomial fusion layer that models the joint representation of the encodings by a higher-order polynomial, with the parameters modelled by a tensor decomposition. We demonstrate the the suitability of this approach through experiments on generated videos evaluated on a range of metrics on video quality, audiovisual synchronisation and generation of blinks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/28/2023

UniFLG: Unified Facial Landmark Generator from Text or Speech

Talking face generation has been extensively investigated owing to its w...
research
06/14/2019

Realistic Speech-Driven Facial Animation with GANs

Speech-driven facial animation is the process that automatically synthes...
research
05/23/2018

End-to-End Speech-Driven Facial Animation with Temporal GANs

Speech-driven facial animation is the process which uses speech signals ...
research
09/20/2023

FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion

Speech-driven 3D facial animation synthesis has been a challenging task ...
research
05/27/2020

Modality Dropout for Improved Performance-driven Talking Faces

We describe our novel deep learning approach for driving animated faces ...
research
06/10/2020

Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging

While deep learning technologies are now capable of generating realistic...
research
07/27/2022

A Hybrid Deep Animation Codec for Low-bitrate Video Conferencing

Deep generative models, and particularly facial animation schemes, can b...

Please sign up or login with your details

Forgot password? Click here to reset