Learning Prior Feature and Attention Enhanced Image Inpainting

08/03/2022
by   Chenjie Cao, et al.
9

Many recent inpainting works have achieved impressive results by leveraging Deep Neural Networks (DNNs) to model various prior information for image restoration. Unfortunately, the performance of these methods is largely limited by the representation ability of vanilla Convolutional Neural Networks (CNNs) backbones.On the other hand, Vision Transformers (ViT) with self-supervised pre-training have shown great potential for many visual recognition and object detection tasks. A natural question is whether the inpainting task can be greatly benefited from the ViT backbone? However, it is nontrivial to directly replace the new backbones in inpainting networks, as the inpainting is an inverse problem fundamentally different from the recognition tasks. To this end, this paper incorporates the pre-training based Masked AutoEncoder (MAE) into the inpainting model, which enjoys richer informative priors to enhance the inpainting process. Moreover, we propose to use attention priors from MAE to make the inpainting model learn more long-distance dependencies between masked and unmasked regions. Sufficient ablations have been discussed about the inpainting and the self-supervised pre-training models in this paper. Besides, experiments on both Places2 and FFHQ demonstrate the effectiveness of our proposed model. Codes and pre-trained models are released in https://github.com/ewrfcas/MAE-FAR.

READ FULL TEXT

page 2

page 11

page 12

page 21

page 22

page 23

page 24

page 25

research
02/02/2021

Test-Time Adaptation for Out-of-distributed Image Inpainting

Deep learning-based image inpainting algorithms have shown great perform...
research
02/28/2023

Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors

Current popular backbones in computer vision, such as Vision Transformer...
research
02/18/2023

Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognition

Recent years have witnessed a boom in self-supervised learning (SSL) in ...
research
06/07/2022

Wavelet Prior Attention Learning in Axial Inpainting Network

Image inpainting is the task of filling masked or unknown regions of an ...
research
11/17/2022

Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information

To effectively exploit the potential of large-scale models, various pre-...
research
03/02/2022

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding

Image inpainting has made significant advances in recent years. However,...
research
06/26/2021

In-N-Out: Towards Good Initialization for Inpainting and Outpainting

In computer vision, recovering spatial information by filling in masked ...

Please sign up or login with your details

Forgot password? Click here to reset