DeepAI
Log In Sign Up

Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse

07/19/2022
by   Sachit Menon, et al.
2

Variational autoencoders (VAEs) suffer from posterior collapse, where the powerful neural networks used for modeling and inference optimize the objective without meaningfully using the latent representation. We introduce inference critics that detect and incentivize against posterior collapse by requiring correspondence between latent variables and the observations. By connecting the critic's objective to the literature in self-supervised contrastive representation learning, we show both theoretically and empirically that optimizing inference critics increases the mutual information between observations and latents, mitigating posterior collapse. This approach is straightforward to implement and requires significantly less training time than prior methods, yet obtains competitive results on three established datasets. Overall, the approach lays the foundation to bridge the previously disconnected frameworks of contrastive learning and probabilistic modeling with variational autoencoders, underscoring the benefits both communities may find at their intersection.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/01/2020

Mutual Information Constraints for Monte-Carlo Objectives

A common failure mode of density models trained as variational autoencod...
06/07/2017

InfoVAE: Information Maximizing Variational Autoencoders

It has been previously observed that variational autoencoders tend to ig...
07/23/2019

Noise Contrastive Variational Autoencoders

We take steps towards understanding the "posterior collapse (PC)" diffic...
03/21/2021

Self-supervised Representation Learning with Relative Predictive Coding

This paper introduces Relative Predictive Coding (RPC), a new contrastiv...
11/30/2022

Variational Laplace Autoencoders

Variational autoencoders employ an amortized inference model to approxim...
02/09/2022

Covariate-informed Representation Learning with Samplewise Optimal Identifiable Variational Autoencoders

Recently proposed identifiable variational autoencoder (iVAE, Khemakhem ...
03/04/2020

Contrastive estimation reveals topic posterior information to linear models

Contrastive learning is an approach to representation learning that util...