DeepAI AI Chat
Log In Sign Up

Deep Directed Generative Autoencoders

by   Sherjil Ozair, et al.

For discrete data, the likelihood P(x) can be rewritten exactly and parametrized into P(X = x) = P(X = x | H = f(x)) P(H = f(x)) if P(X | H) has enough capacity to put no probability mass on any x' for which f(x')≠ f(x), where f(·) is a deterministic discrete function. The log of the first factor gives rise to the log-likelihood reconstruction error of an autoencoder with f(·) as the encoder and P(X|H) as the (probabilistic) decoder. The log of the second term can be seen as a regularizer on the encoded activations h=f(x), e.g., as in sparse autoencoders. Both encoder and decoder can be represented by a deep neural network and trained to maximize the average of the optimal log-likelihood p(x). The objective is to learn an encoder f(·) that maps X to f(X) that has a much simpler distribution than X itself, estimated by P(H). This "flattens the manifold" or concentrates probability mass in a smaller number of (relevant) dimensions over which the distribution factorizes. Generating samples from the model is straightforward using ancestral sampling. One challenge is that regular back-propagation cannot be used to obtain the gradient on the parameters of the encoder, but we find that using the straight-through estimator works well here. We also find that although optimizing a single level of such architecture may be difficult, much better results can be obtained by pre-training and stacking them, gradually transforming the data distribution into one that is more easily captured by a simple parametric model.


page 2

page 8


On the Quantitative Analysis of Decoder-Based Generative Models

The past several years have seen remarkable progress in generative model...

Closing the gap: Exact maximum likelihood training of generative autoencoders using invertible layers

In this work, we provide an exact likelihood alternative to the variatio...

Boltzmann Encoded Adversarial Machines

Restricted Boltzmann Machines (RBMs) are a class of generative neural ne...

Deep Log-Likelihood Ratio Quantization

In this work, a deep learning-based method for log-likelihood ratio (LLR...

Shallow Encoder Deep Decoder (SEDD) Networks for Image Encryption and Decryption

This paper explores a new framework for lossy image encryption and decry...

AQuaMaM: An Autoregressive, Quaternion Manifold Model for Rapidly Estimating Complex SO(3) Distributions

Accurately modeling complex, multimodal distributions is necessary for o...

Deep Neural Networks for the Sequential Probability Ratio Test on Non-i.i.d. Data Series

Classifying sequential data as early as and as accurately as possible is...