Synthesizing Programs for Images using Reinforced Adversarial Learning

04/03/2018
by   Yaroslav Ganin, et al.
0

Advances in deep generative networks have led to impressive results in recent years. Nevertheless, such models can often waste their capacity on the minutiae of datasets, presumably due to weak inductive biases in their decoders. This is where graphics engines may come in handy since they abstract away low-level details and represent images as high-level programs. Current methods that combine deep learning and renderers are limited by hand-crafted likelihood or distance functions, a need for large amounts of supervision, or difficulties in scaling their inference algorithms to richer datasets. To mitigate these issues, we present SPIRAL, an adversarially trained agent that generates a program which is executed by a graphics engine to interpret and sample images. The goal of this agent is to fool a discriminator network that distinguishes between real and rendered data, trained with a distributed reinforcement learning setup without any supervision. A surprising finding is that using the discriminator's output as a reward signal is the key to allow the agent to make meaningful progress at matching the desired output rendering. To the best of our knowledge, this is the first demonstration of an end-to-end, unsupervised and adversarial inverse graphics agent on challenging real world (MNIST, Omniglot, CelebA) and synthetic 3D datasets.

READ FULL TEXT

page 7

page 8

research
03/02/2021

Dual Reinforcement-Based Specification Generation for Image De-Rendering

Advances in deep learning have led to promising progress in inferring gr...
research
03/11/2019

Stroke-based Artistic Rendering Agent with Deep Reinforcement Learning

Excellent painters can use only a few strokes to create a fantastic pain...
research
07/31/2020

Blending Generative Adversarial Image Synthesis with Rendering for Computer Graphics

Conventional computer graphics pipelines require detailed 3D models, mes...
research
10/18/2018

Unsupervised Domain Adaptation for Learning Eye Gaze from a Million Synthetic Images: An Adversarial Approach

With contemporary advancements of graphics engines, recent trend in deep...
research
06/29/2013

Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs

The idea of computer vision as the Bayesian inverse problem to computer ...
research
05/31/2017

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

Researchers have developed excellent feed-forward models that learn to m...
research
10/02/2019

Unsupervised Doodling and Painting with Improved SPIRAL

We investigate using reinforcement learning agents as generative models ...

Please sign up or login with your details

Forgot password? Click here to reset