DeepAI AI Chat
Log In Sign Up

Cross-Modal Contrastive Learning for Text-to-Image Generation

by   Han Zhang, et al.

The output of text-to-image synthesis systems should be coherent, clear, photo-realistic scenes with high semantic fidelity to their conditioned text descriptions. Our Cross-Modal Contrastive Generative Adversarial Network (XMC-GAN) addresses this challenge by maximizing the mutual information between image and text. It does this via multiple contrastive losses which capture inter-modality and intra-modality correspondences. XMC-GAN uses an attentional self-modulation generator, which enforces strong text-image correspondence, and a contrastive discriminator, which acts as a critic as well as a feature encoder for contrastive learning. The quality of XMC-GAN's output is a major step up from previous models, as we show on three challenging datasets. On MS-COCO, not only does XMC-GAN improve state-of-the-art FID from 24.70 to 9.33, but–more importantly–people prefer XMC-GAN by 77.3 for image quality and 74.1 for image-text alignment, compared to three other recent models. XMC-GAN also generalizes to the challenging Localized Narratives dataset (which has longer, more detailed descriptions), improving state-of-the-art FID from 48.70 to 14.12. Lastly, we train and evaluate XMC-GAN on the challenging Open Images data, establishing a strong benchmark FID score of 26.91.


page 7

page 15

page 16

page 17

page 18

page 19


Vision-Language Pre-Training with Triple Contrastive Learning

Vision-language representation learning largely benefits from image-text...

Variational Hetero-Encoder Randomized Generative Adversarial Networks for Joint Image-Text Modeling

For bidirectional joint image-text modeling, we develop variational hete...

A Novel Approach to Artistic Textual Visualization via GAN

While the visualization of statistical data tends to a mature technology...

Contrast-PLC: Contrastive Learning for Packet Loss Concealment

Packet loss concealment (PLC) is challenging in concealing missing conte...

Discrete Contrastive Diffusion for Cross-Modal and Conditional Generation

Diffusion probabilistic models (DPMs) have become a popular approach to ...

Enhancing GAN-Based Vocoders with Contrastive Learning Under Data-limited Condition

Vocoder models have recently achieved substantial progress in generating...

Cycle Text-To-Image GAN with BERT

We explore novel approaches to the task of image generation from their r...

Code Repositories