Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently

12/11/2017
by   Yaodong Yu, et al.
0

We propose a family of nonconvex optimization algorithms that are able to save gradient and negative curvature computations to a large extent, and are guaranteed to find an approximate local minimum with improved runtime complexity. At the core of our algorithms is the division of the entire domain of the objective function into small and large gradient regions: our algorithms only perform gradient descent based procedure in the large gradient region, and only perform negative curvature descent in the small gradient region. Our novel analysis shows that the proposed algorithms can escape the small gradient region in only one negative curvature descent step whenever they enter it, and thus they only need to perform at most N_ϵ negative curvature direction computations, where N_ϵ is the number of times the algorithms enter small gradient regions. For both deterministic and stochastic settings, we show that the proposed algorithms can potentially beat the state-of-the-art local minima finding algorithms. For the finite-sum setting, our algorithm can also outperform the best algorithm in a certain regime.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/18/2017

Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima

We propose stochastic optimization algorithms that can find local minima...
research
10/04/2022

Zeroth-Order Negative Curvature Finding: Escaping Saddle Points without Gradients

We consider escaping saddle points of nonconvex problems where only the ...
research
06/22/2018

Finding Local Minima via Stochastic Nested Variance Reduction

We propose two algorithms that can find local minima faster than the sta...
research
01/27/2022

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O(ε^-7/4) Complexity

This paper studies the accelerated gradient descent for general nonconve...
research
11/17/2017

Neon2: Finding Local Minima via First-Order Oracles

We propose a reduction for non-convex optimization that can (1) turn a s...
research
02/01/2019

Passed & Spurious: analysing descent algorithms and local minima in spiked matrix-tensor model

In this work we analyse quantitatively the interplay between the loss la...
research
03/27/2023

Probing optimisation in physics-informed neural networks

A novel comparison is presented of the effect of optimiser choice on the...

Please sign up or login with your details

Forgot password? Click here to reset