Gradient Estimation for Binary Latent Variables via Gradient Variance Clipping

08/12/2022
by   Russell Z. Kunes, et al.
8

Gradient estimation is often necessary for fitting generative models with discrete latent variables, in contexts such as reinforcement learning and variational autoencoder (VAE) training. The DisARM estimator (Yin et al. 2020; Dong, Mnih, and Tucker 2020) achieves state of the art gradient variance for Bernoulli latent variable models in many contexts. However, DisARM and other estimators have potentially exploding variance near the boundary of the parameter space, where solutions tend to lie. To ameliorate this issue, we propose a new gradient estimator bitflip-1 that has lower variance at the boundaries of the parameter space. As bitflip-1 has complementary properties to existing estimators, we introduce an aggregated estimator, unbiased gradient variance clipping (UGC) that uses either a bitflip-1 or a DisARM gradient update for each coordinate. We theoretically prove that UGC has uniformly lower variance than DisARM. Empirically, we observe that UGC achieves the optimal value of the optimization objectives in toy experiments, discrete VAE training, and in a best subset selection problem.

READ FULL TEXT

page 6

page 10

page 14

page 20

research
11/09/2021

Double Control Variates for Gradient Estimation in Discrete Latent Variable Models

Stochastic gradient-based optimisation for discrete latent variable mode...
research
06/15/2021

Coupled Gradient Estimators for Discrete Latent Variables

Training models with discrete latent variables is challenging due to the...
research
06/15/2022

Training Discrete Deep Generative Models via Gapped Straight-Through Estimator

While deep generative models have succeeded in image processing, natural...
research
03/21/2017

REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

Learning in models with discrete latent variables is challenging due to ...
research
11/04/2016

Reparameterization trick for discrete variables

Low-variance gradient estimation is crucial for learning directed graphi...
research
06/10/2019

Variance Reduction in Gradient Exploration for Online Learning to Rank

Online Learning to Rank (OL2R) algorithms learn from implicit user feedb...
research
05/26/2018

Revisiting Reweighted Wake-Sleep

Discrete latent-variable models, while applicable in a variety of settin...

Please sign up or login with your details

Forgot password? Click here to reset