Finding Approximate Local Minima Faster than Gradient Descent

by   Naman Agarwal, et al.

We design a non-convex second-order optimization algorithm that is guaranteed to return an approximate local minimum in time which scales linearly in the underlying dimension and the number of training examples. The time complexity of our algorithm to find an approximate local minimum is even faster than that of gradient descent to find a critical point. Our algorithm applies to a general class of optimization problems including training a neural network and other non-convex objectives arising in machine learning.


page 1

page 2

page 3

page 4


Escaping Saddle Points for Zeroth-order Nonconvex Optimization using Estimated Gradient Descent

Gradient descent and its variants are widely used in machine learning. H...

Qualitatively characterizing neural network optimization problems

Training neural networks involves solving large-scale non-convex optimiz...

Learning Unitaries by Gradient Descent

We study the hardness of learning unitary transformations by performing ...

On the saddle point problem for non-convex optimization

A central challenge to many fields of science and engineering involves m...

Efficient approaches for escaping higher order saddle points in non-convex optimization

Local search heuristics for non-convex optimizations are popular in appl...

Learn2Hop: Learned Optimization on Rough Landscapes

Optimization of non-convex loss surfaces containing many local minima re...

Exact asymptotic characterisation of running time for approximate gradient descent on random graphs

In this work we study the time complexity for the search of local minima...