Variational Variance: Simple and Reliable Predictive Variance Parameterization

06/08/2020
by   Andrew Stirn, et al.
4

An often overlooked sleight of hand performed with variational autoencoders (VAEs), which has proliferated the literature, is to misrepresent the posterior predictive (decoder) distribution's expectation as a sample from that distribution. Jointly modeling the mean and variance for a normal predictive distribution can result in fragile optimization where the ultimately learned parameters can be ineffective at generating realistic samples. The two most common principled methods to avoid this problem are to either fix the variance or use the single-parameter Bernoulli distribution–both have drawbacks, however. Unfortunately, the problem of jointly optimizing mean and variance networks affects not only unsupervised modeling of continuous data (a taxonomy for many VAE applications) but also regression tasks. To date, only a handful of papers have attempted to resolve these difficulties. In this article, we propose an alternative and attractively simple solution: treat predictive variance variationally. Our approach synergizes with existing VAE-specific theoretical results and, being probabilistically principled, provides access to Empirical Bayes and other such techniques that utilize the observed data to construct well-informed priors. We extend the VAMP prior, which assumes a uniform mixture, by inferring mixture proportions and assignments. This extension amplifies our ability to accurately capture heteroscedastic variance. Notably, our methods experimentally outperform existing techniques on supervised and unsupervised modeling of continuous data.

READ FULL TEXT

page 8

page 9

page 18

research
10/18/2020

Addressing Variance Shrinkage in Variational Autoencoders using Quantile Regression

Estimation of uncertainty in deep learning models is of vital importance...
research
11/25/2019

Improving VAE generations of multimodal data through data-dependent conditional priors

One of the major shortcomings of variational autoencoders is the inabili...
research
02/17/2021

Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE

Variational autoencoders (VAEs) often suffer from posterior collapse, wh...
research
09/20/2021

Deep Quantile Regression for Uncertainty Estimation in Unsupervised and Supervised Lesion Detection

Despite impressive state-of-the-art performance on a wide variety of mac...
research
07/19/2016

Stochastic Backpropagation through Mixture Density Distributions

The ability to backpropagate stochastic gradients through continuous lat...
research
06/23/2020

Simple and Effective VAE Training with Calibrated Decoders

Variational autoencoders (VAEs) provide an effective and simple method f...
research
08/23/2022

String-based Molecule Generation via Multi-decoder VAE

In this paper, we investigate the problem of string-based molecular gene...

Please sign up or login with your details

Forgot password? Click here to reset