Network Fusion for Content Creation with Conditional INNs

05/27/2020
by   Robin Rombach, et al.
3

Artificial Intelligence for Content Creation has the potential to reduce the amount of manual content creation work significantly. While automation of laborious work is welcome, it is only useful if it allows users to control aspects of the creative process when desired. Furthermore, widespread adoption of semi-automatic content creation depends on low barriers regarding the expertise, computational budget and time required to obtain results and experiment with new techniques. With state-of-the-art approaches relying on task-specific models, multi-GPU setups and weeks of training time, we must find ways to reuse and recombine them to meet these requirements. Instead of designing and training methods for controllable content creation from scratch, we thus present a method to repurpose powerful, existing models for new tasks, even though they have never been designed for them. We formulate this problem as a translation between expert models, which includes common content creation scenarios, such as text-to-image and image-to-image translation, as a special case. As this translation is ambiguous, we learn a generative model of hidden representations of one expert conditioned on hidden representations of the other expert. Working on the level of hidden representations makes optimal use of the computational effort that went into the training of the expert model to produce these efficient, low-dimensional representations. Experiments demonstrate that our approach can translate from BERT, a state-of-the-art expert for text, to BigGAN, a state-of-the-art expert for images, to enable text-to-image generation, which neither of the experts can perform on its own. Additional experiments show the wide applicability of our approach across different conditional image synthesis tasks and improvements over existing methods for image modifications.

READ FULL TEXT

page 1

page 3

page 7

page 8

research
07/27/2022

Vector Quantized Image-to-Image Translation

Current image-to-image translation methods formulate the task with condi...
research
03/08/2023

Creating Informal Learning and First Responder Training XR Experiences with the ImmersiveDeck

In recent years eXtended Reality (XR) technologies have matured and have...
research
05/29/2023

Conditional Score Guidance for Text-Driven Image-to-Image Translation

We present a novel algorithm for text-driven image-to-image translation ...
research
03/28/2018

Image Generation and Translation with Disentangled Representations

Generative models have made significant progress in the tasks of modelin...
research
08/10/2020

Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach

Manipulating visual attributes of images through human-written text is a...
research
08/11/2023

DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity

The unprecedented photorealistic results achieved by recent text-to-imag...
research
01/07/2021

The Road to Ubiquitous Personal Fabrication: Modeling-Free Instead of Increasingly Simple

The tools for personal digital fabrication (DF) are on the verge of reac...

Please sign up or login with your details

Forgot password? Click here to reset