Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

by   Yossi Arjevani, et al.

We design an algorithm which finds an ϵ-approximate stationary point (with ∇ F(x)<ϵ) using O(ϵ^-3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p> 2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ϵ,γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.


page 1

page 2

page 3

page 4


Lower Bounds for Non-Convex Stochastic Optimization

We lower bound the complexity of finding ϵ-stationary points (with gradi...

SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator

In this paper, we propose a new technique named Stochastic Path-Integrat...

On the Second-order Convergence Properties of Random Search Methods

We study the theoretical convergence properties of random-search methods...

A Stochastic Tensor Method for Non-convex Optimization

We present a stochastic optimization method that uses a fourth-order reg...

Escaping Saddle Points in Constrained Optimization

In this paper, we focus on escaping from saddle points in smooth nonconv...

SNAP: Finding Approximate Second-Order Stationary Solutions Efficiently for Non-convex Linearly Constrained Problems

This paper proposes low-complexity algorithms for finding approximate se...

Application of a Second-order Stochastic Optimization Algorithm for Fitting Stochastic Epidemiological Models

Epidemiological models have tremendous potential to forecast disease bur...