Lower Bounds for Non-Convex Stochastic Optimization

12/05/2019
by   Yossi Arjevani, et al.
0

We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ^-4 queries to find an ϵ stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ^-3 queries, establishing the optimality of recently proposed variance reduction techniques.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/24/2020

Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

We design an algorithm which finds an ϵ-approximate stationary point (wi...
06/14/2022

Lazy Queries Can Reduce Variance in Zeroth-order Optimization

A major challenge of applying zeroth-order (ZO) methods is the high quer...
07/04/2018

SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator

In this paper, we propose a new technique named Stochastic Path-Integrat...
04/04/2018

Tight Query Complexity Lower Bounds for PCA via Finite Sample Deformed Wigner Law

We prove a query complexity lower bound for approximating the top r dime...
05/05/2021

Randomized Stochastic Variance-Reduced Methods for Stochastic Bilevel Optimization

In this paper, we consider non-convex stochastic bilevel optimization (S...
02/25/2020

Biased Stochastic Gradient Descent for Conditional Stochastic Optimization

Conditional Stochastic Optimization (CSO) covers a variety of applicatio...
02/01/2020

Oracle lower bounds for stochastic gradient sampling algorithms

We consider the problem of sampling from a strongly log-concave density ...