New Benchmarks for Accountable Text-based Visual Re-creation

03/10/2023
by   Zhiwei Zhang, et al.
0

Given a command, humans can directly execute the action after thinking or choose to reject it, with reasonable feedback at the same time. However, the behavior of existing text-to-image generation methods are uncontrollable and irresponsible. In this paper, we construct extensive experiments to verify whether they can be accountable (say no and explain why) for those prohibited instructions. To this end, we define a novel text-based visual re-creation task and construct new synthetic CLEVR-NOT dataset (620K) and manually pictured Fruit-NOT dataset (50K). In our method, one text-image pair as the query is fed into the machine, and the model gives a yes or no answer after visual and textual reasoning. If the answer is yes, the image auto-encoder and auto-regressive transformer must complete the visual re-creation under the premise of ensuring image quality, otherwise the system needs to explain why the commands cannot be completed or prohibited. We provide a detailed analysis of experimental results in image quality, answer accuracy, and model behavior in the face of uncertainty and imperfect user queries. Our results demonstrate the difficulty of a single model for both textual and visual reasoning. We also hope our explorations and findings can bring valuable insights about the accountability of text-based image generation models. Code and datasets can be found at https://matrix-alpha.github.io.

READ FULL TEXT

page 2

page 3

page 5

page 6

page 7

page 13

research
12/07/2022

Judge, Localize, and Edit: Ensuring Visual Commonsense Morality for Text-to-Image Generation

Text-to-image generation methods produce high-resolution and high-qualit...
research
04/14/2023

AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics

Recent advancements in language-image models have led to the development...
research
02/08/2022

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

Generating images from textual descriptions has gained a lot of attentio...
research
08/09/2023

TextPainter: Multimodal Text Image Generation withVisual-harmony and Text-comprehension for Poster Design

Text design is one of the most critical procedures in poster design, as ...
research
09/05/2023

Breaking Barriers to Creative Expression: Co-Designing and Implementing an Accessible Text-to-Image Interface

Text-to-image generation models have grown in popularity due to their ab...
research
06/11/2023

Face0: Instantaneously Conditioning a Text-to-Image Model on a Face

We present Face0, a novel way to instantaneously condition a text-to-ima...
research
03/19/2022

ALAP-AE: As-Lite-as-Possible Auto-Encoder

We present a novel algorithm to reduce tensor compute required by a cond...

Please sign up or login with your details

Forgot password? Click here to reset