Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

by   Yingchen Yu, et al.

Image inpainting is an underdetermined inverse problem, it naturally allows diverse contents that fill up the missing or corrupted regions reasonably and realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in transformers is suboptimal as corrupted regions can have arbitrary shapes with contexts from arbitrary directions. We propose BAT-Fill, an image inpainting framework with a novel bidirectional autoregressive transformer (BAT) that models deep bidirectional contexts for autoregressive generation of diverse inpainting contents. BAT-Fill inherits the merits of transformers and CNNs in a two-stage manner, which allows to generate high-resolution contents without being constrained by the quadratic complexity of attention in transformers. Specifically, it first generates pluralistic image structures of low resolution by adapting transformers and then synthesizes realistic texture details of high resolutions with a CNN-based up-sampling network. Extensive experiments over multiple datasets show that BAT-Fill achieves superior diversity and fidelity in image inpainting qualitatively and quantitatively.


page 1

page 4

page 7

page 8


MAT: Mask-Aware Transformer for Large Hole Image Inpainting

Recent studies have shown the importance of modeling long-range interact...

High-Fidelity Pluralistic Image Completion with Transformers

Image completion has made tremendous progress with convolutional neural ...

Region-aware Attention for Image Inpainting

Recent attention-based image inpainting methods have made inspiring prog...

FT-TDR: Frequency-guided Transformer and Top-Down Refinement Network for Blind Face Inpainting

Blind face inpainting refers to the task of reconstructing visual conten...

Wavelet Prior Attention Learning in Axial Inpainting Network

Image inpainting is the task of filling masked or unknown regions of an ...

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding

Image inpainting has made significant advances in recent years. However,...

ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

We present ASSET, a neural architecture for automatically modifying an i...