Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network

11/16/2017
by   Avisek Lahiri, et al.
0

Contemporary benchmark methods for image inpainting are based on deep generative models and specifically leverage adversarial loss for yielding realistic reconstructions. However, these models cannot be directly applied on image/video sequences because of an intrinsic drawback- the reconstructions might be independently realistic, but, when visualized as a sequence, often lacks fidelity to the original uncorrupted sequence. The fundamental reason is that these methods try to find the best matching latent space representation near to natural image manifold without any explicit distance based loss. In this paper, we present a semantically conditioned Generative Adversarial Network (GAN) for sequence inpainting. The conditional information constrains the GAN to map a latent representation to a point in image manifold respecting the underlying pose and semantics of the scene. To the best of our knowledge, this is the first work which simultaneously addresses consistency and correctness of generative model based inpainting. We show that our generative model learns to disentangle pose and appearance information; this independence is exploited by our model to generate highly consistent reconstructions. The conditional information also aids the generator network in GAN to produce sharper images compared to the original GAN formulation. This helps in achieving more appealing inpainting performance. Though generic, our algorithm was targeted for inpainting on faces. When applied on CelebA and Youtube Faces datasets, the proposed method results in a significant improvement over the current benchmark, both in terms of quantitative evaluation (Peak Signal to Noise Ratio) and human visual scoring over diversified combinations of resolutions and deformations.

READ FULL TEXT

page 13

page 14

page 15

page 16

page 17

page 18

page 19

page 20

research
10/20/2018

Improved Techniques for GAN based Facial Inpainting

In this paper we present several architectural and optimization recipes ...
research
05/13/2021

Extreme Face Inpainting with Sketch-Guided Conditional GAN

Recovering badly damaged face images is a useful yet challenging task, e...
research
12/18/2019

Unsupervised Adversarial Image Inpainting

We consider inpainting in an unsupervised setting where there is neither...
research
08/16/2019

The Angel is in the Priors: Improving GAN based Image and Sequence Inpainting with Better Noise and Structural Priors

Contemporary deep learning based inpainting algorithms are mainly based ...
research
10/18/2021

Continuation of Famous Art with AI: A Conditional Adversarial Network Inpainting Approach

Much of the state-of-the-art in image synthesis inspired by real artwork...
research
09/06/2023

Hierarchical-level rain image generative model based on GAN

Autonomous vehicles are exposed to various weather during operation, whi...
research
05/23/2020

AnimGAN: A Spatiotemporally-Conditioned Generative Adversarial Network for Character Animation

Producing realistic character animations is one of the essential tasks i...

Please sign up or login with your details

Forgot password? Click here to reset