Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

04/26/2021
by   Yingchen Yu, et al.
11

Image inpainting is an underdetermined inverse problem, it naturally allows diverse contents that fill up the missing or corrupted regions reasonably and realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in transformers is suboptimal as corrupted regions can have arbitrary shapes with contexts from arbitrary directions. We propose BAT-Fill, an image inpainting framework with a novel bidirectional autoregressive transformer (BAT) that models deep bidirectional contexts for autoregressive generation of diverse inpainting contents. BAT-Fill inherits the merits of transformers and CNNs in a two-stage manner, which allows to generate high-resolution contents without being constrained by the quadratic complexity of attention in transformers. Specifically, it first generates pluralistic image structures of low resolution by adapting transformers and then synthesizes realistic texture details of high resolutions with a CNN-based up-sampling network. Extensive experiments over multiple datasets show that BAT-Fill achieves superior diversity and fidelity in image inpainting qualitatively and quantitatively.

READ FULL TEXT

page 1

page 4

page 7

page 8

research
03/29/2022

MAT: Mask-Aware Transformer for Large Hole Image Inpainting

Recent studies have shown the importance of modeling long-range interact...
research
03/25/2021

High-Fidelity Pluralistic Image Completion with Transformers

Image completion has made tremendous progress with convolutional neural ...
research
04/03/2022

Region-aware Attention for Image Inpainting

Recent attention-based image inpainting methods have made inspiring prog...
research
03/18/2023

Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction

Predicting panoramic indoor lighting from a single perspective image is ...
research
07/23/2021

WaveFill: A Wavelet-based Generation Network for Image Inpainting

Image inpainting aims to complete the missing or corrupted regions of im...
research
06/07/2022

Wavelet Prior Attention Learning in Axial Inpainting Network

Image inpainting is the task of filling masked or unknown regions of an ...
research
03/02/2022

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding

Image inpainting has made significant advances in recent years. However,...

Please sign up or login with your details

Forgot password? Click here to reset