Generative Memory-Guided Semantic Reasoning Model for Image Inpainting

10/01/2021
by   Xin Feng, et al.
0

Most existing methods for image inpainting focus on learning the intra-image priors from the known regions of the current input image to infer the content of the corrupted regions in the same image. While such methods perform well on images with small corrupted regions, it is challenging for these methods to deal with images with large corrupted area due to two potential limitations: 1) such methods tend to overfit each single training pair of images relying solely on the intra-image prior knowledge learned from the limited known area; 2) the inter-image prior knowledge about the general distribution patterns of visual semantics, which can be transferred across images sharing similar semantics, is not exploited. In this paper, we propose the Generative Memory-Guided Semantic Reasoning Model (GM-SRM), which not only learns the intra-image priors from the known regions, but also distills the inter-image reasoning priors to infer the content of the corrupted regions. In particular, the proposed GM-SRM first pre-learns a generative memory from the whole training data to capture the semantic distribution patterns in a global view. Then the learned memory are leveraged to retrieve the matching inter-image priors for the current corrupted image to perform semantic reasoning during image inpainting. While the intra-image priors are used for guaranteeing the pixel-level content consistency, the inter-image priors are favorable for performing high-level semantic reasoning, which is particularly effective for inferring semantic content for large corrupted area. Extensive experiments on Paris Street View, CelebA-HQ, and Places2 benchmarks demonstrate that our GM-SRM outperforms the state-of-the-art methods for image inpainting in terms of both the visual quality and quantitative metrics.

READ FULL TEXT

page 1

page 2

page 4

page 7

page 9

page 10

page 12

research
07/26/2016

Semantic Image Inpainting with Deep Generative Models

Semantic image inpainting is a challenging task where large missing regi...
research
04/06/2022

ShowFace: Coordinated Face Inpainting with Memory-Disentangled Refinement Networks

Face inpainting aims to complete the corrupted regions of the face image...
research
07/29/2022

Reference-Guided Texture and Structure Inference for Image Inpainting

Existing learning-based image inpainting methods are still in challenge ...
research
12/25/2017

Deep Blind Image Inpainting

Image inpainting is a challenging problem as it needs to fill the inform...
research
06/20/2023

TransRef: Multi-Scale Reference Embedding Transformer for Reference-Guided Image Inpainting

Image inpainting for completing complicated semantic environments and di...
research
03/29/2022

In-N-Out Generative Learning for Dense Unsupervised Video Segmentation

In this paper, we focus on the unsupervised Video Object Segmentation (V...
research
02/11/2019

Semantic Hierarchical Priors for Intrinsic Image Decomposition

Intrinsic Image Decomposition (IID) is a challenging and interesting com...

Please sign up or login with your details

Forgot password? Click here to reset