Blending Generative Adversarial Image Synthesis with Rendering for Computer Graphics

07/31/2020
by   Ekim Yurtsever, et al.
0

Conventional computer graphics pipelines require detailed 3D models, meshes, textures, and rendering engines to generate 2D images from 3D scenes. These processes are labor-intensive. We introduce Hybrid Neural Computer Graphics (HNCG) as an alternative. The contribution is a novel image formation strategy to reduce the 3D model and texture complexity of computer graphics pipelines. Our main idea is straightforward: Given a 3D scene, render only important objects of interest and use generative adversarial processes for synthesizing the rest of the image. To this end, we propose a novel image formation strategy to form 2D semantic images from 3D scenery consisting of simple object models without textures. These semantic images are then converted into photo-realistic RGB images with a state-of-the-art conditional Generative Adversarial Network (cGAN) based image synthesizer trained on real-world data. Meanwhile, objects of interest are rendered using a physics-based graphics engine. This is necessary as we want to have full control over the appearance of objects of interest. Finally, the partially-rendered and cGAN synthesized images are blended with a blending GAN. We show that the proposed framework outperforms conventional rendering with ablation and comparison studies. Semantic retention and Fréchet Inception Distance (FID) measurements were used as the main performance metrics.

READ FULL TEXT

page 6

page 7

page 14

page 15

page 17

research
01/10/2018

Instance Map based Image Synthesis with a Denoising Generative Adversarial Network

Semantic layouts based Image synthesizing, which has benefited from the ...
research
04/08/2020

State of the Art on Neural Rendering

Efficient rendering of photo-realistic virtual worlds is a long standing...
research
06/17/2022

Texture Generation Using Graph Generative Adversarial Network And Differentiable Rendering

Novel texture synthesis for existing 3D mesh models is an important step...
research
06/18/2020

Neural Graphics Pipeline for Controllable Image Generation

We present Neural Graphics Pipeline (NGP), a hybrid generative model tha...
research
04/03/2018

Synthesizing Programs for Images using Reinforced Adversarial Learning

Advances in deep generative networks have led to impressive results in r...
research
01/19/2019

Synthesizing facial photometries and corresponding geometries using generative adversarial networks

Artificial data synthesis is currently a well studied topic with useful ...
research
01/09/2018

SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

Synthesizing realistic images from human drawn sketches is a challenging...

Please sign up or login with your details

Forgot password? Click here to reset