Learning to Have an Ear for Face Super-Resolution

09/27/2019
by   Givi Meishvili, et al.
0

We propose a novel method to perform extreme (16x) face super-resolution by exploiting audio. Super-resolution is the task of recovering a high-resolution image from a low-resolution one. When the resolution of the input image is too low (e.g., 8x8 pixels), the loss of information is so dire that the details of the original identity have been lost. However, when the low-resolution image is extracted from a video, the audio track is also available. Because the audio carries information about the face identity, we propose to exploit it in the face reconstruction process. Towards this goal, we propose a model and a training procedure to extract information about the identity of a person from her audio track and to combine it with the information extracted from the low-resolution input image, which relates more to pose and colors of the face. We demonstrate that the combination of these two inputs yields high-resolution images that better capture the correct identity of the face. In particular, we show that audio can assist in recovering attributes such as the gender and the identity, and thus improve the correctness of the image reconstruction process. Our procedure does not make use of human annotation and thus can be easily trained with existing video datasets. Moreover, we show that our model allows one to mix low-resolution images and audio from different videos and to generate realistic faces with semantically meaningful combinations.

READ FULL TEXT

page 6

page 8

research
03/26/2019

Verification of Very Low-Resolution Faces Using An Identity-Preserving Deep Face Super-Resolution Network

Face super-resolution methods usually aim at producing visually appealin...
research
09/16/2020

Multiple Exemplars-based Hallucinationfor Face Super-resolution and Editing

Given a really low-resolution input image of a face (say 16x16 or 8x8 pi...
research
11/06/2018

Super-Identity Convolutional Neural Network for Face Hallucination

Face hallucination is a generative task to super-resolve the facial imag...
research
11/20/2021

Identity-Preserving Pose-Robust Face Hallucination Through Face Subspace Prior

Over the past few decades, numerous attempts have been made to address t...
research
08/17/2022

Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors

In this paper, we explore an interesting question of what can be obtaine...
research
05/30/2021

Identity and Attribute Preserving Thumbnail Upscaling

We consider the task of upscaling a low resolution thumbnail image of a ...
research
03/23/2016

Global-Local Face Upsampling Network

Face hallucination, which is the task of generating a high-resolution fa...

Please sign up or login with your details

Forgot password? Click here to reset