Noise Contrastive Variational Autoencoders

07/23/2019
by   Octavian-Eugen Ganea, et al.
3

We take steps towards understanding the "posterior collapse (PC)" difficulty in variational autoencoders (VAEs), i.e. a degenerate optimum in which the latent codes become independent of their corresponding inputs. We rely on calculus of variations and theoretically explore a few popular VAE models, showing that PC always occurs for non-parametric encoders and decoders. Inspired by the popular noise contrastive estimation algorithm, we propose NC-VAE where the encoder discriminates between the latent codes of real data and of some artificially generated noise, in addition to encouraging good data reconstruction abilities. Theoretically, we prove that our model cannot reach PC and provide novel lower bounds. Our method is straightforward to implement and has the same run-time as vanilla VAE. Empirically, we showcase its benefits on popular image and text datasets.

READ FULL TEXT

page 21

page 22

research
09/06/2023

CR-VAE: Contrastive Regularization on Variational Autoencoders for Preventing Posterior Collapse

The Variational Autoencoder (VAE) is known to suffer from the phenomenon...
research
02/12/2019

Contrastive Variational Autoencoder Enhances Salient Features

Variational autoencoders are powerful algorithms for identifying dominan...
research
12/06/2022

Three Variations on Variational Autoencoders

Variational autoencoders (VAEs) are one class of generative probabilisti...
research
02/16/2019

WiSE-VAE: Wide Sample Estimator VAE

Variational Auto-encoders (VAEs) have been very successful as methods fo...
research
04/04/2023

Fully Variational Noise-Contrastive Estimation

By using the underlying theory of proper scoring rules, we design a fami...
research
07/19/2022

Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse

Variational autoencoders (VAEs) suffer from posterior collapse, where th...
research
05/18/2020

Learning and Inference in Imaginary Noise Models

Inspired by recent developments in learning smoothed densities with empi...

Please sign up or login with your details

Forgot password? Click here to reset