Visual Prompt Tuning for Generative Transfer Learning

10/03/2022
by   Kihyuk Sohn, et al.
0

Transferring knowledge from an image synthesis model trained on a large dataset is a promising direction for learning generative image models from various domains efficiently. While previous works have studied GAN models, we present a recipe for learning vision transformers by generative knowledge transfer. We base our framework on state-of-the-art generative vision transformers that represent an image as a sequence of visual tokens to the autoregressive or non-autoregressive transformers. To adapt to a new domain, we employ prompt tuning, which prepends learnable tokens called prompt to the image token sequence, and introduce a new prompt design for our task. We study on a variety of visual domains, including visual task adaptation benchmark <cit.>, with varying amount of training images, and show effectiveness of knowledge transfer and a significantly better image generation quality over existing works.

READ FULL TEXT

page 20

page 22

page 23

page 25

page 26

page 29

page 31

page 34

research
09/09/2022

Improved Masked Image Generation with Token-Critic

Non-autoregressive generative transformers recently demonstrated impress...
research
03/07/2023

Lformer: Text-to-Image Generation with L-shape Block Parallel Decoding

Generative transformers have shown their superiority in synthesizing hig...
research
04/14/2023

M2T: Masking Transformers Twice for Faster Decoding

We show how bidirectional transformers trained for masked token predicti...
research
03/27/2023

Learning Expressive Prompting With Residuals for Vision Transformers

Prompt learning is an efficient approach to adapt transformers by insert...
research
03/01/2023

StraIT: Non-autoregressive Generation with Stratified Image Transformer

We propose Stratified Image Transformer(StraIT), a pure non-autoregressi...
research
05/27/2022

MixMIM: Mixed and Masked Image Modeling for Efficient Visual Representation Learning

In this study, we propose Mixed and Masked Image Modeling (MixMIM), a si...
research
02/08/2022

MaskGIT: Masked Generative Image Transformer

Generative transformers have experienced rapid popularity growth in the ...

Please sign up or login with your details

Forgot password? Click here to reset