Minimax Efficient Finite-Difference Stochastic Gradient Estimators Using Black-Box Function Evaluations

07/08/2020 ∙ by Henry Lam, et al. ∙ 0 ∙ share

We consider stochastic gradient estimation using noisy black-box function evaluations. A standard approach is to use the finite-difference method or its variants. While natural, it is open to our knowledge whether its statistical accuracy is the best possible. This paper argues so by showing that central finite-difference is a nearly minimax optimal zeroth-order gradient estimator, among both the class of linear estimators and the much larger class of all (nonlinear) estimators.

READ FULL TEXT

Information-Theoretic Lower Bounds for Zero-Order Stochastic Gradient Estimation

In this paper we analyze the necessary number of samples to estimate the...
03/31/2020 ∙ by Abdulrahman Alabdulkareem, et al. ∙ 0 ∙ share

read it
On the Application of Danskin's Theorem to Derivative-Free Minimax Optimization

Motivated by Danskin's theorem, gradient-based methods have been applied...
05/15/2018 ∙ by Abdullah Al-Dujaili, et al. ∙ 0 ∙ share

read it
Provable Gradient Variance Guarantees for Black-Box Variational Inference

Recent variational inference methods use stochastic gradient estimators ...
06/19/2019 ∙ by Justin Domke, et al. ∙ 0 ∙ share

read it
Low-variance Black-box Gradient Estimates for the Plackett-Luce Distribution

Learning models with discrete latent variables using stochastic gradient...
11/22/2019 ∙ by Artyom Gadetsky, et al. ∙ 0 ∙ share

read it
Sequential Learning of Active Subspaces

In recent years, active subspace methods (ASMs) have become a popular me...
07/26/2019 ∙ by Nathan Wycoff, et al. ∙ 0 ∙ share

read it
Rao-Blackwellized Stochastic Gradients for Discrete Distributions

We wish to compute the gradient of an expectation over a finite or count...
10/10/2018 ∙ by Runjing Liu, et al. ∙ 0 ∙ share

read it
Enhanced Balancing of Bias-Variance Tradeoff in Stochastic Estimation: A Minimax Perspective

Biased stochastic estimators, such as finite-differences for noisy gradi...
02/12/2019 ∙ by Henry Lam, et al. ∙ 0 ∙ share

read it

Minimax Efficient Finite-Difference Stochastic Gradient Estimators Using Black-Box Function Evaluations

Authors

Information-Theoretic Lower Bounds for Zero-Order Stochastic Gradient Estimation

On the Application of Danskin's Theorem to Derivative-Free Minimax Optimization

Provable Gradient Variance Guarantees for Black-Box Variational Inference

Low-variance Black-box Gradient Estimates for the Plackett-Luce Distribution

Sequential Learning of Active Subspaces

Rao-Blackwellized Stochastic Gradients for Discrete Distributions

Enhanced Balancing of Bias-Variance Tradeoff in Stochastic Estimation: A Minimax Perspective

This week in AI