Log In Sign Up

Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse

by   Sachit Menon, et al.

Variational autoencoders (VAEs) suffer from posterior collapse, where the powerful neural networks used for modeling and inference optimize the objective without meaningfully using the latent representation. We introduce inference critics that detect and incentivize against posterior collapse by requiring correspondence between latent variables and the observations. By connecting the critic's objective to the literature in self-supervised contrastive representation learning, we show both theoretically and empirically that optimizing inference critics increases the mutual information between observations and latents, mitigating posterior collapse. This approach is straightforward to implement and requires significantly less training time than prior methods, yet obtains competitive results on three established datasets. Overall, the approach lays the foundation to bridge the previously disconnected frameworks of contrastive learning and probabilistic modeling with variational autoencoders, underscoring the benefits both communities may find at their intersection.


page 1

page 2

page 3

page 4


Mutual Information Constraints for Monte-Carlo Objectives

A common failure mode of density models trained as variational autoencod...

InfoVAE: Information Maximizing Variational Autoencoders

It has been previously observed that variational autoencoders tend to ig...

Noise Contrastive Variational Autoencoders

We take steps towards understanding the "posterior collapse (PC)" diffic...

Self-supervised Representation Learning with Relative Predictive Coding

This paper introduces Relative Predictive Coding (RPC), a new contrastiv...

Variational Laplace Autoencoders

Variational autoencoders employ an amortized inference model to approxim...

Covariate-informed Representation Learning with Samplewise Optimal Identifiable Variational Autoencoders

Recently proposed identifiable variational autoencoder (iVAE, Khemakhem ...

Contrastive estimation reveals topic posterior information to linear models

Contrastive learning is an approach to representation learning that util...