Finding Approximate Local Minima Faster than Gradient Descent

11/03/2016
by   Naman Agarwal, et al.
0

We design a non-convex second-order optimization algorithm that is guaranteed to return an approximate local minimum in time which scales linearly in the underlying dimension and the number of training examples. The time complexity of our algorithm to find an approximate local minimum is even faster than that of gradient descent to find a critical point. Our algorithm applies to a general class of optimization problems including training a neural network and other non-convex objectives arising in machine learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/03/2019

Escaping Saddle Points for Zeroth-order Nonconvex Optimization using Estimated Gradient Descent

Gradient descent and its variants are widely used in machine learning. H...
12/19/2014

Qualitatively characterizing neural network optimization problems

Training neural networks involves solving large-scale non-convex optimiz...
01/31/2020

Learning Unitaries by Gradient Descent

We study the hardness of learning unitary transformations by performing ...
05/19/2014

On the saddle point problem for non-convex optimization

A central challenge to many fields of science and engineering involves m...
02/18/2016

Efficient approaches for escaping higher order saddle points in non-convex optimization

Local search heuristics for non-convex optimizations are popular in appl...
07/20/2021

Learn2Hop: Learned Optimization on Rough Landscapes

Optimization of non-convex loss surfaces containing many local minima re...
05/10/2021

Exact asymptotic characterisation of running time for approximate gradient descent on random graphs

In this work we study the time complexity for the search of local minima...