An Information-Theoretic Analysis of Deep Latent-Variable Models

by   Alexander A. Alemi, et al.

We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.


Learning multi-modal generative models with permutation-invariant encoders and tighter variational bounds

Devising deep latent variable models for multi-modal data has been a lon...

The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models

A variety of learning objectives have been proposed for training latent ...

Hierarchical Disentangled Representations

Deep latent-variable models learn representations of high-dimensional da...

Trading Information between Latents in Hierarchical Variational Autoencoders

Variational Autoencoders (VAEs) were originally motivated (Kingma We...

A General Method for Amortizing Variational Filtering

We introduce the variational filtering EM algorithm, a simple, general-p...

Benefits of Overparameterization in Single-Layer Latent Variable Generative Models

One of the most surprising and exciting discoveries in supervising learn...

Fundamental Limitations of Control and Filtering in Continuous-Time Systems: An Information-Theoretic Analysis

While information-theoretic methods have been introduced to investigate ...

Please sign up or login with your details

Forgot password? Click here to reset