Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures

10/29/2019
by   Wu Lin, et al.
0

Stein's method (Stein, 1973; 1981) is a powerful tool for statistical applications, and has had a significant impact in machine learning. Stein's lemma plays an essential role in Stein's method. Previous applications of Stein's lemma either required strong technical assumptions or were limited to Gaussian distributions with restricted covariance structures. In this work, we extend Stein's lemma to exponential-family mixture distributions including Gaussian distributions with full covariance structures. Our generalization enables us to establish a connection between Stein's lemma and the reparamterization trick to derive gradients of expectations of a large class of functions under weak assumptions. Using this connection, we can derive many new reparameterizable gradient-identities that goes beyond the reach of existing works. For example, we give gradient identities when expectation is taken with respect to Student's t-distribution, skew Gaussian, exponentially modified Gaussian, and normal inverse Gaussian.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/25/2017

Expectation Propagation for t-Exponential Family Using Q-Algebra

Exponential family distributions are highly useful in machine learning s...
research
05/11/2021

A General Derivative Identity for the Conditional Expectation with Focus on the Exponential Family

Consider a pair of random vectors (𝐗,𝐘) and the conditional expectation ...
research
01/09/2018

Quadrature Compound: An approximating family of distributions

Compound distributions allow construction of a rich set of distributions...
research
03/13/2020

The Elliptical Processes: a New Family of Flexible Stochastic Processes

We present the elliptical processes-a new family of stochastic processes...
research
12/29/2016

Quantum Clustering and Gaussian Mixtures

The mixture of Gaussian distributions, a soft version of k-means , is co...
research
03/04/2020

Maximal Causes for Exponential Family Observables

The data model of standard sparse coding assumes a weighted linear summa...
research
01/12/2022

On the Statistical Complexity of Sample Amplification

Given n i.i.d. samples drawn from an unknown distribution P, when is it ...

Please sign up or login with your details

Forgot password? Click here to reset