GumBolt: Extending Gumbel trick to Boltzmann priors

05/18/2018
by   Amir H. Khoshaman, et al.
0

Boltzmann machines (BMs) are appealing candidates for powerful priors in variational autoencoders (VAEs), as they are capable of capturing nontrivial and multi-modal distributions over discrete variables. However, indifferentiability of the discrete units prohibits using the reparameterization trick, essential for low-noise back propagation. The Gumbel trick resolves this problem in a consistent way by relaxing the variables and distributions, but it is incompatible with BM priors. Here, we propose the GumBolt, a model that extends the Gumbel trick to BM priors in VAEs. GumBolt is significantly simpler than the recently proposed methods with BM prior and outperforms them by a considerable margin. It achieves state-of-the-art performance on permutation invariant MNIST and OMNIGLOT datasets in the scope of models with only discrete latent variables. Moreover, the performance can be further improved by allowing multi-sampled (importance-weighted) estimation of log-likelihood in training, which was not possible with previous models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/18/2018

DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors

Boltzmann machines are powerful distributions that have been shown to be...
research
02/14/2018

DVAE++: Discrete Variational Autoencoders with Overlapping Transformations

Training of discrete latent variable models remains challenging because ...
research
09/07/2016

Discrete Variational Autoencoders

Probabilistic models with discrete latent variables naturally capture da...
research
09/01/2023

Learning multi-modal generative models with permutation-invariant encoders and tighter variational bounds

Devising deep latent variable models for multi-modal data has been a lon...
research
12/01/2016

Piecewise Latent Variables for Neural Variational Text Processing

Advances in neural variational inference have facilitated the learning o...
research
02/20/2017

Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors

Restricted Boltzmann Machines are described by the Gibbs measure of a bi...
research
05/25/2023

Revisiting Structured Variational Autoencoders

Structured variational autoencoders (SVAEs) combine probabilistic graphi...

Please sign up or login with your details

Forgot password? Click here to reset