The Variational InfoMax Learning Objective

03/07/2020

∙

Bayesian Inference and Information Bottleneck are the two most popular objectives for neural networks, but they can be optimised only via a variational lower bound: the Variational Information Bottleneck (VIB). In this manuscript we show that the two objectives are actually equivalent to the InfoMax: maximise the information between the data and the labels. The InfoMax representation of the two objectives is not relevant only per se, since it helps to understand the role of the network capacity, but also because it allows us to derive a variational objective, the Variational InfoMax (VIM), that maximises them directly without resorting to any lower bound. The theoretical improvement of VIM over VIB is highlighted by the computational experiments, where the model trained by VIM improves the VIB model in three different tasks: accuracy, robustness to noise and representation quality.

READ FULL TEXT

The Variational InfoMax Learning Objective

Variational Predictive Information Bottleneck

Relevant sparse codes with variational information bottleneck

The Variational Deficiency Bottleneck

On the failure of variational score matching for VAE models

The variational infomax autoencoder

Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding

Learning Representations in Reinforcement Learning:An Information Bottleneck Approach

The Variational InfoMax Learning Objective

Related Research

Variational Predictive Information Bottleneck

Relevant sparse codes with variational information bottleneck

The Variational Deficiency Bottleneck

On the failure of variational score matching for VAE models

The variational infomax autoencoder

Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding

Learning Representations in Reinforcement Learning:An Information Bottleneck Approach