Closed Form Variances for Variational Auto-Encoders

by   Graham Fyffe, et al.

We propose a reformulation of Variational Auto-Encoders eliminating half of the network outputs (the variances) in a deep network setting. While it is well known that the posterior is in general intractable, we show that the variances of Gaussian posteriors and likelihoods may be solved in closed form, producing improved variational lower bounds over their learned counterparts in experiments. The closed forms reduce to remarkably simple expressions – in particular, one optimal choice for the posterior variance is simply the identity matrix. We arrive at these conclusions by analyzing the variational lower bound objective irrespective of any particular network architecture, deriving its partial derivatives and closed form solutions for all parameters but the posterior means. In deriving the closed form likelihood variance, we show that the objective is underdetermined, which we resolve by constraining the presumed information content of the data examples. Any of these modifications may be applied to simplify, and perhaps improve, any Variational Auto-Encoder.


page 18

page 19

page 20


Closed Form Variational Objectives For Bayesian Neural Networks with a Single Hidden Layer

In this note we consider setups in which variational objectives for Baye...

Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions

This work develops problem statements related to encoders and autoencode...

The Evidence Lower Bound of Variational Autoencoders Converges to a Sum of Three Entropies

The central objective function of a variational autoencoder (VAE) is its...

Gosper's algorithm and Bell numbers

Computers are good at evaluating finite sums in closed form, but there a...

A closed form scale bound for the (ε, δ)-differentially private Gaussian Mechanism valid for all privacy regimes

The standard closed form lower bound on σ for providing (ϵ, δ)-different...

On the Variance of the Fisher Information for Deep Learning

The Fisher information matrix (FIM) has been applied to the realm of dee...

Closed-form Expressions for Maximum Mean Discrepancy with Applications to Wasserstein Auto-Encoders

The Maximum Mean Discrepancy (MMD) has found numerous applications in st...

Please sign up or login with your details

Forgot password? Click here to reset