RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection

by   Shancong Mou, et al.

Generative adversarial networks (GANs), trained on a large-scale image dataset, can be a good approximator of the natural image manifold. GAN-inversion, using a pre-trained generator as a deep generative prior, is a promising tool for image restoration under corruptions. However, the performance of GAN-inversion can be limited by a lack of robustness to unknown gross corruptions, i.e., the restored image might easily deviate from the ground truth. In this paper, we propose a Robust GAN-inversion (RGI) method with a provable robustness guarantee to achieve image restoration under unknown gross corruptions, where a small fraction of pixels are completely corrupted. Under mild assumptions, we show that the restored image and the identified corrupted region mask converge asymptotically to the ground truth. Moreover, we extend RGI to Relaxed-RGI (R-RGI) for generator fine-tuning to mitigate the gap between the GAN learned manifold and the true image manifold while avoiding trivial overfitting to the corrupted input image, which further improves the image restoration and corrupted region mask identification performance. The proposed RGI/R-RGI method unifies two important applications with state-of-the-art (SOTA) performance: (i) mask-free semantic inpainting, where the corruptions are unknown missing regions, the restored background can be used to restore the missing content; (ii) unsupervised pixel-wise anomaly detection, where the corruptions are unknown anomalous regions, the retrieved mask can be used as the anomalous region's segmentation mask.


Semantic Image Inpainting with Deep Generative Models

Semantic image inpainting is a challenging task where large missing regi...

Pixel-wise Anomaly Detection in Complex Driving Scenes

The inability of state-of-the-art semantic segmentation methods to detec...

MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking

In the past few years, generative models like Generative Adversarial Net...

Digital Cardan Grille: A Modern Approach for Information Hiding

In this paper, a new framework for construction of Cardan grille for inf...

High-Fidelity Image Inpainting with GAN Inversion

Image inpainting seeks a semantically consistent way to recover the corr...

Robust Unsupervised StyleGAN Image Restoration

GAN-based image restoration inverts the generative process to repair ima...

On the Anomalous Generalization of GANs

Generative models, especially Generative Adversarial Networks (GANs), ha...

Please sign up or login with your details

Forgot password? Click here to reset