Progressive Learning and Disentanglement of Hierarchical Representations

02/24/2020
by   Zhiyuan Li, et al.
0

Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE). However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised. Motivated by the concept of "starting small", we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions. The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new representations at different levels of abstraction. We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap. We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations. By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

READ FULL TEXT

page 7

page 8

page 11

research
12/21/2019

Learning Representations by Maximizing Mutual Information in Variational Autoencoder

Variational autoencoders (VAE) have ushered in a new era of unsupervised...
research
06/26/2018

Hierarchical VampPrior Variational Fair Auto-Encoder

Decision making is a process that is extremely prone to different biases...
research
11/24/2019

dpVAEs: Fixing Sample Generation for Regularized VAEs

Unsupervised representation learning via generative modeling is a staple...
research
01/19/2022

Online Deep Learning based on Auto-Encoder

Online learning is an important technical means for sketching massive re...
research
12/04/2021

Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach

The existence of a universal learning architecture in human cognition is...
research
09/03/2019

Improving Disentangled Representation Learning with the Beta Bernoulli Process

To improve the ability of VAE to disentangle in the latent space, existi...
research
10/31/2018

Deep Generative Model with Beta Bernoulli Process for Modeling and Learning Confounding Factors

While deep representation learning has become increasingly capable of se...

Please sign up or login with your details

Forgot password? Click here to reset