IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning

04/02/2022
by   Zhenhuan Liu, et al.
0

Conditional image generation is an active research topic including text2image and image translation. Recently image manipulation with linguistic instruction brings new challenges of multimodal conditional generation. However, traditional conditional image generation models mainly focus on generating high-quality and visually realistic images, and lack resolving the partial consistency between image and instruction. To address this issue, we propose an Increment Reasoning Generative Adversarial Network (IR-GAN), which aims to reason the consistency between visual increment in images and semantic increment in instructions. First, we introduce the word-level and instruction-level instruction encoders to learn user's intention from history-correlated instructions as semantic increment. Second, we embed the representation of semantic increment into that of source image for generating target image, where source image plays the role of referring auxiliary. Finally, we propose a reasoning discriminator to measure the consistency between visual increment and semantic increment, which purifies user's intention and guarantees the good logic of generated target image. Extensive experiments and visualization conducted on two datasets show the effectiveness of IR-GAN.

READ FULL TEXT

page 2

page 7

page 8

research
02/23/2018

Interactive Image Manipulation with Natural Language Instruction Commands

We propose an interactive image-manipulation system with natural languag...
research
06/17/2019

Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification

In this paper, we address multimodal language understanding for unconstr...
research
03/14/2019

MirrorGAN: Learning Text-to-image Generation by Redescription

Generating an image from a given text description has two goals: visual ...
research
09/25/2019

Conditional Transferring Features: Scaling GANs to Thousands of Classes with 30

Generative adversarial network (GAN) has greatly improved the quality of...
research
08/02/2023

ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation

While language-guided image manipulation has made remarkable progress, t...
research
06/11/2018

A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructions

This paper focuses on a multimodal language understanding method for car...
research
11/26/2022

Target-Free Text-guided Image Manipulation

We tackle the problem of target-free text-guided image manipulation, whi...

Please sign up or login with your details

Forgot password? Click here to reset