Log In Sign Up

Demystifying Inductive Biases for β-VAE Based Architectures

by   Dominik Zietlow, et al.

The performance of β-Variational-Autoencoders (β-VAEs) and their variants on learning semantically meaningful, disentangled representations is unparalleled. On the other hand, there are theoretical arguments suggesting the impossibility of unsupervised disentanglement. In this work, we shed light on the inductive bias responsible for the success of VAE-based architectures. We show that in classical datasets the structure of variance, induced by the generating factors, is conveniently aligned with the latent directions fostered by the VAE objective. This builds the pivotal bias on which the disentangling abilities of VAEs rely. By small, elaborate perturbations of existing datasets, we hide the convenient correlation structure that is easily exploited by a variety of architectures. To demonstrate this, we construct modified versions of standard datasets in which (i) the generative factors are perfectly preserved; (ii) each image undergoes a mild transformation causing a small change of variance; (iii) the leading VAE-based disentanglement architectures fail to produce disentangled representations whilst the performance of a non-variational method remains unchanged. The construction of our modifications is nontrivial and relies on recent progress on mechanistic understanding of β-VAEs and their connection to PCA. We strengthen that connection by providing additional insights that are of stand-alone interest.


page 6

page 7

page 16

page 17


Understanding disentangling in β-VAE

We present new intuitions and theoretical assessments of the emergence o...

GroupifyVAE: from Group-based Definition to VAE-based Unsupervised Representation Disentanglement

The key idea of the state-of-the-art VAE-based unsupervised representati...

WeLa-VAE: Learning Alternative Disentangled Representations Using Weak Labels

Learning disentangled representations without supervision or inductive b...

Variational Autoencoders Pursue PCA Directions (by Accident)

The Variational Autoencoder (VAE) is a powerful architecture capable of ...

A Closer Look at Disentangling in β-VAE

In many data analysis tasks, it is beneficial to learn representations w...

Re-parameterizing VAEs for stability

We propose a theoretical approach towards the training numerical stabili...

Early Visual Concept Learning with Unsupervised Deep Learning

Automated discovery of early visual concepts from raw image data is a ma...