DeepAI AI Chat
Log In Sign Up

ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

by   Wanrong Zhu, et al.

Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with the text references. This is different from human language processing, for which visual imaginations often improve comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of CLIP and DALL-E, two cross-modal models pre-trained on large-scale image-text pairs, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding imagination with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics' correlations with human similarity judgments in many circumstances.


page 6

page 7

page 8

page 16

page 17

page 18

page 19


BERTScore: Evaluating Text Generation with BERT

We propose BERTScore, an automatic evaluation metric for text generation...

Translation between Molecules and Natural Language

Joint representations between images and text have been deeply investiga...

Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training

Automatic generation of natural language from images has attracted exten...

CAMERA: A Multimodal Dataset and Benchmark for Ad Text Generation

In response to the limitations of manual online ad production, significa...

Keep it Neutral: Using Natural Language Inference to Improve Generation

We explore incorporating natural language inference (NLI) into the text ...

Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity

Automatic Audio Captioning (AAC) refers to the task of translating an au...

Few-Shot Natural Language Generation by Rewriting Templates

Virtual assistants such as Google Assistant, Alexa and Siri enable users...