DeepAI
Log In Sign Up

f-Divergence Variational Inference

09/28/2020
by   Neng Wan, et al.
19

This paper introduces the f-divergence variational inference (f-VI) that generalizes variational inference to all f-divergences. Initiated from minimizing a crafty surrogate f-divergence that shares the statistical consistency with the f-divergence, the f-VI framework not only unifies a number of existing VI methods, e.g. Kullback-Leibler VI, Rényi's α-VI, and χ-VI, but offers a standardized toolkit for VI subject to arbitrary divergences from f-divergence family. A general f-variational bound is derived and provides a sandwich estimate of marginal likelihood (or evidence). The development of the f-VI unfolds with a stochastic optimization scheme that utilizes the reparameterization trick, importance weighting and Monte Carlo approximation; a mean-field approximation scheme that generalizes the well-known coordinate ascent variational inference (CAVI) is also proposed for f-VI. Empirical examples, including variational autoencoders and Bayesian neural networks, are provided to demonstrate the effectiveness and the wide applicability of f-VI.

READ FULL TEXT
02/06/2016

Rényi Divergence Variational Inference

This paper introduces the variational Rényi bound (VR) that extends trad...
06/22/2012

Fast Variational Inference in the Conjugate Exponential Family

We present a general method for deriving collapsed variational inference...
05/02/2018

Alpha-Beta Divergence For Variational Inference

This paper introduces a variational approximation framework using direct...
04/01/2021

Variational Inference MPC using Tsallis Divergence

In this paper, we provide a generalized framework for Variational Infere...
02/19/2018

Distribution Matching in Variational Inference

The difficulties in matching the latent posterior to the prior, balancin...
03/02/2020

Bayesian Neural Networks With Maximum Mean Discrepancy Regularization

Bayesian Neural Networks (BNNs) are trained to optimize an entire distri...
11/04/2021

Variational Inference with Holder Bounds

The recent introduction of thermodynamic integration techniques has prov...