DVAE++: Discrete Variational Autoencoders with Overlapping Transformations

02/14/2018
by   Arash Vahdat, et al.
0

Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

05/18/2018

DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors

Boltzmann machines are powerful distributions that have been shown to be...
09/07/2016

Discrete Variational Autoencoders

Probabilistic models with discrete latent variables naturally capture da...
05/18/2018

GumBolt: Extending Gumbel trick to Boltzmann priors

Boltzmann machines (BMs) are appealing candidates for powerful priors in...
02/23/2018

Learning Latent Permutations with Gumbel-Sinkhorn Networks

Permutations and matchings are core building blocks in a variety of late...
08/05/2021

Sparse Communication via Mixed Distributions

Neural networks and other machine learning models compute continuous rep...
11/27/2020

Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents

Discrete latent variables are considered important for real world data, ...
04/01/2021

Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication

Neural networks and other machine learning models compute continuous rep...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.