Speech2Face: Learning the Face Behind a Voice

05/23/2019
by   Tae-Hyun Oh, et al.
22

How much can we infer about a person's looks from the way they speak? In this paper, we study the task of reconstructing a facial image of a person from a short audio recording of that person speaking. We design and train a deep neural network to perform this task using millions of natural Internet/YouTube videos of people speaking. During training, our model learns voice-face correlations that allow it to produce images that capture various physical attributes of the speakers such as age, gender and ethnicity. This is done in a self-supervised manner, by utilizing the natural co-occurrence of faces and speech in Internet videos, without the need to model attributes explicitly. We evaluate and numerically quantify how–and in what manner–our Speech2Face reconstructions, obtained directly from audio, resemble the true face images of the speakers.

READ FULL TEXT

page 1

page 2

page 4

page 5

page 6

page 7

page 8

page 9

research
05/25/2019

Reconstructing faces from voices

Voice profiling aims at inferring various human parameters from their sp...
research
02/27/2023

Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech

The goal of this work is zero-shot text-to-speech synthesis, with speaki...
research
12/19/2018

Physical Attribute Prediction Using Deep Residual Neural Networks

Images taken from the Internet have been used alongside Deep Learning fo...
research
04/09/2019

Crossmodal Voice Conversion

Humans are able to imagine a person's voice from the person's appearance...
research
04/30/2020

APB2Face: Audio-guided face reenactment with auxiliary pose and blink signals

Audio-guided face reenactment aims at generating photorealistic faces us...
research
11/23/2022

Whose Emotion Matters? Speaker Detection without Prior Knowledge

The task of emotion recognition in conversations (ERC) benefits from the...
research
06/06/2020

The Criminality From Face Illusion

The automatic analysis of face images can generate predictions about a p...

Please sign up or login with your details

Forgot password? Click here to reset