Improved Transformer for High-Resolution GANs

by   Long Zhao, et al.

Attention-based models, exemplified by the Transformer, can effectively model long range dependency, but suffer from the quadratic complexity of self-attention operation, making them difficult to be adopted for high-resolution image generation based on Generative Adversarial Networks (GANs). In this paper, we introduce two key ingredients to Transformer to address this challenge. First, in low-resolution stages of the generative process, standard global self-attention is replaced with the proposed multi-axis blocked self-attention which allows efficient mixing of local and global attention. Second, in high-resolution stages, we drop self-attention while only keeping multi-layer perceptrons reminiscent of the implicit neural function. To further improve the performance, we introduce an additional self-modulation component based on cross-attention. The resulting model, denoted as HiT, has a linear computational complexity with respect to the image size and thus directly scales to synthesizing high definition images. We show in the experiments that the proposed HiT achieves state-of-the-art FID scores of 31.87 and 2.95 on unconditional ImageNet 128 × 128 and FFHQ 256 × 256, respectively, with a reasonable throughput. We believe the proposed HiT is an important milestone for generators in GANs which are completely free of convolutions.


page 6

page 8

page 17

page 18


Self-Attention Generative Adversarial Networks

In this paper, we propose the Self-Attention Generative Adversarial Netw...

MaxViT: Multi-Axis Vision Transformer

Transformers have recently gained significant attention in the computer ...

Global Self-Attention Networks for Image Recognition

Recently, a series of works in computer vision have shown promising resu...

Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?

In this paper we address the problem of fine-tuned text generation with ...

Deep is a Luxury We Don't Have

Medical images come in high resolutions. A high resolution is vital for ...

Image Transformer

Image generation has been successfully cast as an autoregressive sequenc...

Image Tranformer

Image generation has been successfully cast as an autoregressive sequenc...

Code Repositories


Refactoring dalle-pytorch and taming-transformers for TPU VM

view repo