Learning Representations by Maximizing Mutual Information in Variational Autoencoder

12/21/2019
by   Ali Lotfi Rezaabad, et al.
0

Variational autoencoders (VAE) have ushered in a new era of unsupervised learning methods for complex distributions. Although these techniques are elegant in their approach, they are typically not useful for representation learning. In this work, we propose a simple yet powerful class of VAEs that simultaneously result in meaningful learned representations. Our solution is to combine traditional VAEs with mutual information maximization, with the goal to enhance amortized inference in VAEs using Information Theoretic techniques. We call this approach InfoMax-VAE, and such an approach can significantly boost the quality of learned high-level representations. We realize this through the explicit maximization of information measures associated with the representation. Using extensive experiments on varied datasets and setups, we show that InfoMax-VAE outperforms contemporary popular approaches, including Info-VAE and β-VAE.

READ FULL TEXT
research
05/28/2020

VMI-VAE: Variational Mutual Information Maximization Framework for VAE With Discrete and Continuous Priors

Variational Autoencoder is a scalable method for learning latent variabl...
research
02/16/2018

Auto-Encoding Total Correlation Explanation

Advances in unsupervised learning enable reconstruction and generation o...
research
05/27/2019

Wyner VAE: Joint and Conditional Generation with Succinct Common Representation Learning

A new variational autoencoder (VAE) model is proposed that learns a succ...
research
02/24/2020

Progressive Learning and Disentanglement of Hierarchical Representations

Learning rich representation from data is an important task for deep gen...
research
07/06/2021

InfoNCE is a variational autoencoder

We show that a popular self-supervised learning method, InfoNCE, is a sp...
research
01/25/2019

Unsupervised speech representation learning using WaveNet autoencoders

We consider the task of unsupervised extraction of meaningful latent rep...
research
08/20/2018

Learning deep representations by mutual information estimation and maximization

Many popular representation-learning algorithms use training objectives ...

Please sign up or login with your details

Forgot password? Click here to reset