Gradient Descent Converges to Minimizers

02/16/2016
by   Jason D. Lee, et al.
0

We show that gradient descent converges to a local minimizer, almost surely with random initialization. This is proved by applying the Stable Manifold Theorem from dynamical systems theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/07/2019

Distributed Gradient Descent: Nonconvergence to Saddle Points and the Stable-Manifold Theorem

The paper studies a distributed gradient descent (DGD) process and consi...
research
10/20/2017

First-order Methods Almost Always Avoid Saddle Points

We establish that first-order methods avoid saddle points for almost all...
research
07/10/2020

Learning Unstable Dynamical Systems with Time-Weighted Logarithmic Loss

When training the parameters of a linear dynamical model, the gradient d...
research
08/03/2022

Gradient descent provably escapes saddle points in the training of shallow ReLU networks

Dynamical systems theory has recently been applied in optimization to pr...
research
09/16/2016

Gradient Descent Learns Linear Dynamical Systems

We prove that gradient descent efficiently converges to the global optim...
research
06/01/2018

Implicit Bias of Gradient Descent on Linear Convolutional Networks

We show that gradient descent on full-width linear convolutional network...
research
12/04/2019

Exponential convergence of Sobolev gradient descent for a class of nonlinear eigenproblems

We propose to use the Łojasiewicz inequality as a general tool for analy...

Please sign up or login with your details

Forgot password? Click here to reset