Dual Reinforcement-Based Specification Generation for Image De-Rendering

by   Ramakanth Pasunuru, et al.

Advances in deep learning have led to promising progress in inferring graphics programs by de-rendering computer-generated images. However, current methods do not explore which decoding methods lead to better inductive bias for inferring graphics programs. In our work, we first explore the effectiveness of LSTM-RNN versus Transformer networks as decoders for order-independent graphics programs. Since these are sequence models, we must choose an ordering of the objects in the graphics programs for likelihood training. We found that the LSTM performance was highly sensitive to the sequence ordering (random order vs. pattern-based order), while Transformer performance was roughly independent of the sequence ordering. Further, we present a policy gradient based reinforcement learning approach for better inductive bias in the decoder via multiple diverse rewards based both on the graphics program specification and the rendered image. We also explore the combination of these complementary rewards. We achieve state-of-the-art results on two graphics program generation datasets.


page 1

page 5

page 8


Synthesizing Programs for Images using Reinforced Adversarial Learning

Advances in deep generative networks have led to impressive results in r...

AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning

The performance of the code a compiler generates depends on the order in...

Comprehensive Optimization of Parametric Kernels for Graphics Processing Units

This work deals with the optimization of computer programs targeting Gra...

Learning to Infer Graphics Programs from Hand-Drawn Images

We introduce a model that learns to convert simple hand drawings into gr...

Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis

Program synthesis is the task of automatically generating a program cons...

TSO: Curriculum Generation using continuous optimization

The training of deep learning models poses vast challenges of including ...

Learning to Generate Synthetic 3D Training Data through Hybrid Gradient

Synthetic images rendered by graphics engines are a promising source for...

Please sign up or login with your details

Forgot password? Click here to reset