The Usual Suspects? Reassessing Blame for VAE Posterior Collapse

12/23/2019
by   Bin Dai, et al.
7

In narrow asymptotic settings Gaussian VAE models of continuous data have been shown to possess global optima aligned with ground-truth distributions. Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice. However, contrary to conventional wisdom that largely assigns blame for this phenomena on the undue influence of KL-divergence regularization, we will argue that posterior collapse is, at least in part, a direct consequence of bad local minima inherent to the loss surface of deep autoencoder networks. In particular, we prove that even small nonlinear perturbations of affine VAE decoder models can produce such minima, and in deeper models, analogous minima can force the VAE to behave like an aggressive truncation operator, provably discarding information along all latent dimensions in certain circumstances. Regardless, the underlying message here is not meant to undercut valuable existing explanations of posterior collapse, but rather, to refine the discussion and elucidate alternative risk factors that may have been previously underappreciated.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/14/2018

Variational Autoencoder with Implicit Optimal Priors

The variational autoencoder (VAE) is a powerful generative model that ca...
research
06/08/2023

Unscented Autoencoder

The Variational Autoencoder (VAE) is a seminal approach in deep generati...
research
10/29/2019

Bridging the ELBO and MMD

One of the challenges in training generative models such as the variatio...
research
04/27/2020

A Batch Normalized Inference Network Keeps the KL Vanishing Away

Variational Autoencoder (VAE) is widely used as a generative model to ap...
research
09/13/2019

ρ-VAE: Autoregressive parametrization of the VAE encoder

We make a minimal, but very effective alteration to the VAE model. This ...
research
05/31/2019

On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder

Using powerful posterior distributions is a popular approach to achievin...

Please sign up or login with your details

Forgot password? Click here to reset