Bridging the ELBO and MMD

10/29/2019
by   Talip Ucar, et al.
31

One of the challenges in training generative models such as the variational auto encoder (VAE) is avoiding posterior collapse. When the generator has too much capacity, it is prone to ignoring latent code. This problem is exacerbated when the dataset is small, and the latent dimension is high. The root of the problem is the ELBO objective, specifically the Kullback-Leibler (KL) divergence term in objective function <cit.>. This paper proposes a new objective function to replace the KL term with one that emulates the maximum mean discrepancy (MMD) objective. It also introduces a new technique, named latent clipping, that is used to control distance between samples in latent space. A probabilistic autoencoder model, named μ-VAE, is designed and trained on MNIST and MNIST Fashion datasets, using the new objective function and is shown to outperform models trained with ELBO and β-VAE objective. The μ-VAE is less prone to posterior collapse, and can generate reconstructions and new samples in good quality. Latent representations learned by μ-VAE are shown to be good and can be used for downstream tasks such as classification.

READ FULL TEXT

page 4

page 6

page 7

page 8

page 13

page 14

research
09/30/2019

On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation

Variational Autoencoders (VAEs) are known to suffer from learning uninfo...
research
06/10/2020

Joint Training of Variational Auto-Encoder and Latent Energy-Based Model

This paper proposes a joint training method to learn both the variationa...
research
04/24/2019

Generated Loss and Augmented Training of MNIST VAE

The variational autoencoder (VAE) framework is a popular option for trai...
research
02/09/2022

Covariate-informed Representation Learning with Samplewise Optimal Identifiable Variational Autoencoders

Recently proposed identifiable variational autoencoder (iVAE, Khemakhem ...
research
09/13/2017

Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories

Sketch is an important media for human to communicate ideas, which refle...
research
12/23/2019

The Usual Suspects? Reassessing Blame for VAE Posterior Collapse

In narrow asymptotic settings Gaussian VAE models of continuous data hav...
research
09/30/2021

Towards Better Data Augmentation using Wasserstein Distance in Variational Auto-encoder

VAE, or variational auto-encoder, compresses data into latent attributes...

Please sign up or login with your details

Forgot password? Click here to reset