GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis

by   Ming Tao, et al.

Synthesizing high-fidelity complex images from text is challenging. Based on large pretraining, the autoregressive and diffusion models can synthesize photo-realistic images. Although these large models have shown notable progress, there remain three flaws. 1) These models require tremendous training data and parameters to achieve good performance. 2) The multi-step generation design slows the image synthesis process heavily. 3) The synthesized visual features are difficult to control and require delicately designed prompts. To enable high-quality, efficient, fast, and controllable text-to-image synthesis, we propose Generative Adversarial CLIPs, namely GALIP. GALIP leverages the powerful pretrained CLIP model both in the discriminator and generator. Specifically, we propose a CLIP-based discriminator. The complex scene understanding ability of CLIP enables the discriminator to accurately assess the image quality. Furthermore, we propose a CLIP-empowered generator that induces the visual concepts from CLIP through bridge features and prompts. The CLIP-integrated generator and discriminator boost training efficiency, and as a result, our model only requires about 3 parameters, achieving comparable results to large pretrained autoregressive and diffusion models. Moreover, our model achieves 120 times faster synthesis speed and inherits the smooth latent space from GAN. The extensive experimental results demonstrate the excellent performance of our GALIP. Code is available at


page 3

page 4

page 5

page 6

page 7

page 8

page 9


Wave-U-Net Discriminator: Fast and Lightweight Discriminator for Generative Adversarial Network-Based Speech Synthesis

In speech synthesis, a generative adversarial network (GAN), training a ...

BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion

Recent text-to-image diffusion models have demonstrated an astonishing c...

A Shared Representation for Photorealistic Driving Simulators

A powerful simulator highly decreases the need for real-world tests when...

Cross-View Panorama Image Synthesis

In this paper, we tackle the problem of synthesizing a ground-view panor...

Disentanglement in a GAN for Unconditional Speech Synthesis

Can we develop a model that can synthesize realistic speech directly fro...

IT3D: Improved Text-to-3D Generation with Explicit View Synthesis

Recent strides in Text-to-3D techniques have been propelled by distillin...

IMAGINE: Image Synthesis by Image-Guided Model Inversion

We introduce an inversion based method, denoted as IMAge-Guided model IN...

Please sign up or login with your details

Forgot password? Click here to reset