Monte Carlo Gradient Estimation in Machine Learning

by   Shakir Mohamed, et al.

This paper is a broad and accessible survey of the methods we have at our disposal for Monte Carlo gradient estimation in machine learning and across the statistical sciences: the problem of computing the gradient of an expectation of a function with respect to parameters defining the distribution that is integrated; the problem of sensitivity analysis. In machine learning research, this gradient problem lies at the core of many learning problems, in supervised, unsupervised and reinforcement learning. We will generally seek to rewrite such gradients in a form that allows for Monte Carlo estimation, allowing them to be easily and efficiently used and analysed. We explore three strategies--the pathwise, score function, and measure-valued gradient estimators--exploring their historical developments, derivation, and underlying assumptions. We describe their use in other fields, show how they are related and can be combined, and expand on their possible generalisations. Wherever Monte Carlo gradient estimators have been derived and deployed in the past, important advances have followed. A deeper and more widely-held understanding of this problem will lead to further advances, and it is these advances that we wish to support.


page 1

page 2

page 3

page 4


Quasi-Monte Carlo Variational Inference

Many machine learning problems involve Monte Carlo gradient estimators. ...

Gradient estimators for normalising flows

Recently a machine learning approach to Monte-Carlo simulations called N...

Optimal Learning from the Doob-Dynkin lemma

The Doob-Dynkin Lemma gives conditions on two functions X and Y that ens...

ddml: Double/debiased machine learning in Stata

We introduce the package ddml for Double/Debiased Machine Learning (DDML...

A unified view of likelihood ratio and reparameterization gradients

Reparameterization (RP) and likelihood ratio (LR) gradient estimators ar...

Efficient Estimation of the Local Robustness of Machine Learning Models

Machine learning models often need to be robust to noisy input data. The...

Adaptive Monte-Carlo Optimization

The celebrated Monte Carlo method estimates a quantity that is expensive...

Code Repositories


Machine Learning Tutorials

view repo

Please sign up or login with your details

Forgot password? Click here to reset