Semantic Object Accuracy for Generative Text-to-Image Synthesis

by   Tobias Hinz, et al.

Generative adversarial networks conditioned on simple textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image synthesis models is still challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address the aforementioned challenges we introduce both a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. Our model adds an object pathway to both the generator and the discriminator to explicitly learn features of individual objects. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are specifically mentioned in the image caption, e.g. whether an image generated from "a car driving down the street" contains a car. Our evaluation shows that models which explicitly model individual objects outperform models which only model global image characteristics. However, the SOA also shows that despite this increased performance current models still struggle to generate images that contain realistic objects of multiple different domains.


page 4

page 6

page 9

page 11

page 12

page 13


CookGAN: Meal Image Synthesis from Ingredients

In this work we propose a new computational framework, based on generati...

Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network

This paper presents a novel method to deal with the challenging task of ...

The Art of Food: Meal Image Synthesis from Ingredients

In this work we propose a new computational framework, based on generati...

Text-to-image synthesis method evaluation based on visual patterns

A commonly used evaluation metric for text-to-image synthesis is the Inc...

TISE: A Toolbox for Text-to-Image Synthesis Evaluation

In this paper, we conduct a study on state-of-the-art methods for single...

Generative Guiding Block: Synthesizing Realistic Looking Variants Capable of Even Large Change Demands

Realistic image synthesis is to generate an image that is perceptually i...

Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark

We provide a new multi-task benchmark for evaluating text-to-image model...

Please sign up or login with your details

Forgot password? Click here to reset