Unbiased Multi-Modality Guidance for Image Inpainting

by   Yongsheng Yu, et al.
Institute of Software, Chinese Academy of Sciences
University of the Chinese Academy of Sciences

Image inpainting is an ill-posed problem to recover missing or damaged image content based on incomplete images with masks. Previous works usually predict the auxiliary structures (e.g., edges, segmentation and contours) to help fill visually realistic patches in a multi-stage fashion. However, imprecise auxiliary priors may yield biased inpainted results. Besides, it is time-consuming for some methods to be implemented by multiple stages of complex neural networks. To solve this issue, we develop an end-to-end multi-modality guided transformer network, including one inpainting branch and two auxiliary branches for semantic segmentation and edge textures. Within each transformer block, the proposed multi-scale spatial-aware attention module can learn the multi-modal structural features efficiently via auxiliary denormalization. Different from previous methods relying on direct guidance from biased priors, our method enriches semantically consistent context in an image based on discriminative interplay information from multiple modalities. Comprehensive experiments on several challenging image inpainting datasets show that our method achieves state-of-the-art performance to deal with various regular/irregular masks efficiently.


page 4

page 10

page 11


SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

In this paper, we focus on image inpainting task, aiming at recovering t...

ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors

The image inpainting task fills missing areas of a corrupted image. Desp...

Context-Aware Image Inpainting with Learned Semantic Priors

Recent advances in image inpainting have shown impressive results for ge...

Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid

Restoring reasonable and realistic content for arbitrary missing regions...

DDColor: Towards Photo-Realistic and Semantic-Aware Image Colorization via Dual Decoders

Automatic image colorization is a particularly challenging problem. Due ...

Semi-parametric Image Inpainting

This paper introduces a semi-parametric approach to image inpainting for...

SaiNet: Stereo aware inpainting behind objects with generative networks

In this work, we present an end-to-end network for stereo-consistent ima...

Please sign up or login with your details

Forgot password? Click here to reset