Retrieval-Augmented Diffusion Models

04/25/2022
by   Andreas Blattmann, et al.
0

Generative image synthesis with diffusion models has recently achieved excellent visual quality in several tasks such as text-based or class-conditional image synthesis. Much of this success is due to a dramatic increase in the computational capacity invested in training these models. This work presents an alternative approach: inspired by its successful application in natural language processing, we propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. During training, our diffusion model is trained with similar visual features retrieved via CLIP and from the neighborhood of each training instance. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained, such as class-conditional or text-image synthesis, and can be conditioned on both text and image embeddings. Moreover, we can apply our approach to unconditional generation, where it achieves state-of-the-art performance. Our approach incurs low computational and memory overheads and is easy to implement. We discuss its relationship to concurrent work and will publish code and pretrained models soon.

READ FULL TEXT

page 2

page 3

page 4

page 6

page 9

page 10

page 13

page 14

research
07/26/2022

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

Novel architectures have recently improved generative image synthesis le...
research
06/01/2023

Wuerstchen: Efficient Pretraining of Text-to-Image Models

We introduce Wuerstchen, a novel technique for text-to-image synthesis t...
research
06/01/2022

DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder

Recently most successful image synthesis models are multi stage process ...
research
12/27/2022

Exploring Transformer Backbones for Image Diffusion Models

We present an end-to-end Transformer based Latent Diffusion model for im...
research
06/16/2023

CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models

Recent work has studied text-to-audio synthesis using large amounts of p...
research
02/24/2023

Modulating Pretrained Diffusion Models for Multimodal Image Synthesis

We present multimodal conditioning modules (MCM) for enabling conditiona...
research
02/22/2022

Retrieval Augmented Classification for Long-Tail Visual Recognition

We introduce Retrieval Augmented Classification (RAC), a generic approac...

Please sign up or login with your details

Forgot password? Click here to reset