Distributed Gradient Descent: Nonconvergence to Saddle Points and the Stable-Manifold Theorem

08/07/2019
by   Brian Swenson, et al.
0

The paper studies a distributed gradient descent (DGD) process and considers the problem of showing that in nonconvex optimization problems, DGD typically converges to local minima rather than saddle points. The paper considers unconstrained minimization of a smooth objective function. In centralized settings, the problem of demonstrating nonconvergence to saddle points of gradient descent (and variants) is typically handled by way of the stable-manifold theorem from classical dynamical systems theory. However, the classical stable-manifold theorem is not applicable in distributed settings. The paper develops an appropriate stable-manifold theorem for DGD showing that convergence to saddle points may only occur from a low-dimensional stable manifold. Under appropriate assumptions (e.g., coercivity), this result implies that DGD typically converges to local minima and not to saddle points.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/16/2016

Gradient Descent Converges to Minimizers

We show that gradient descent converges to a local minimizer, almost sur...
research
08/12/2020

Distributed Gradient Flow: Nonsmoothness, Nonconvexity, and Saddle Point Evasion

The paper considers distributed gradient flow (DGF) for multi-agent nonc...
research
02/13/2020

Solution manifold and Its Statistical Applications

A solution manifold is the collection of points in a d-dimensional space...
research
06/26/2019

No Pressure! Addressing the Problem of Local Minima in Manifold Learning Algorithms

Nonlinear embedding manifold learning methods provide invaluable visual ...
research
10/20/2017

First-order Methods Almost Always Avoid Saddle Points

We establish that first-order methods avoid saddle points for almost all...
research
12/19/2019

Central limit theorems for stochastic gradient descent with averaging for stable manifolds

In this article we establish new central limit theorems for Ruppert-Poly...
research
10/12/2021

Global Convergence of Triangularized Orthogonalization-free Method

This paper proves the global convergence of a triangularized orthogonali...

Please sign up or login with your details

Forgot password? Click here to reset