Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

06/24/2020
by   Yossi Arjevani, et al.
0

We design an algorithm which finds an ϵ-approximate stationary point (with ∇ F(x)<ϵ) using O(ϵ^-3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p> 2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ϵ,γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/05/2019

Lower Bounds for Non-Convex Stochastic Optimization

We lower bound the complexity of finding ϵ-stationary points (with gradi...
research
07/04/2018

SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator

In this paper, we propose a new technique named Stochastic Path-Integrat...
research
06/26/2023

Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization

Bilevel optimization has various applications such as hyper-parameter op...
research
02/20/2023

Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks

We consider the problem of minimizing a non-convex objective while prese...
research
10/25/2021

On the Second-order Convergence Properties of Random Search Methods

We study the theoretical convergence properties of random-search methods...
research
11/23/2019

A Stochastic Tensor Method for Non-convex Optimization

We present a stochastic optimization method that uses a fourth-order reg...
research
08/02/2017

Application of a Second-order Stochastic Optimization Algorithm for Fitting Stochastic Epidemiological Models

Epidemiological models have tremendous potential to forecast disease bur...

Please sign up or login with your details

Forgot password? Click here to reset