Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

11/21/2019
by   Ceyuan Yang, et al.
13

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises. In this work, we show that highly-structured semantic hierarchy emerges as variation factors for synthesizing scenes from the generative representations in state-of-the-art GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with a broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent variation factors facilitates semantic scene manipulation.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 8

page 12

page 14

page 15

page 16

page 17

page 19

05/18/2020

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Although Generative Adversarial Networks (GANs) have made significant pr...
07/13/2020

Closed-Form Factorization of Latent Semantics in GANs

A rich set of semantic attributes has been shown to emerge in the latent...
04/26/2021

EigenGAN: Layer-Wise Eigen-Learning for GANs

Recent studies on Generative Adversarial Network (GAN) reveal that diffe...
06/03/2021

Semantic Palette: Guiding Scene Generation with Class Proportions

Despite the recent progress of generative adversarial networks (GANs) at...
10/19/2017

Generative Adversarial Networks: An Overview

Generative adversarial networks (GANs) provide a way to learn deep repre...
07/20/2020

Generative Hierarchical Features from Synthesizing Images

Generative Adversarial Networks (GANs) have recently advanced image synt...
07/19/2018

Compositional GAN: Learning Conditional Image Composition

Generative Adversarial Networks (GANs) can produce images of surprising ...

Code Repositories

higan

[IJCV 2020] Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.