Perturbed gradient descent with occupation time

05/09/2020

∙

This paper develops further the idea of perturbed gradient descent, by adapting perturbation with the history of state via the notation of occupation time for saddle points. The proposed algorithm PGDOT is shown to converge at least as fast as perturbed gradient descent (PGD) algorithm, and is guaranteed to avoid getting stuck at saddle points. The analysis is corroborated by experimental results.

READ FULL TEXT

Perturbed gradient descent with occupation time

A Realistic Example in 2 Dimension that Gradient Descent Takes Exponential Time to Escape Saddle Points

On the Differentially Private Nature of Perturbed Gradient Descent

A frequency-domain analysis of inexact gradient descent

Gradient Descent Can Take Exponential Time to Escape Saddle Points

Fast and Accurate Optimization of Metasurfaces with Gradient Descent and the Woodbury Matrix Identity

A convergence analysis of the perturbed compositional gradient flow: averaging principle and normal deviations

Exit Time Analysis for Approximations of Gradient Descent Trajectories Around Saddle Points

Perturbed gradient descent with occupation time

Related Research

A Realistic Example in 2 Dimension that Gradient Descent Takes Exponential Time to Escape Saddle Points

On the Differentially Private Nature of Perturbed Gradient Descent

A frequency-domain analysis of inexact gradient descent

Gradient Descent Can Take Exponential Time to Escape Saddle Points

Fast and Accurate Optimization of Metasurfaces with Gradient Descent and the Woodbury Matrix Identity

A convergence analysis of the perturbed compositional gradient flow: averaging principle and normal deviations

Exit Time Analysis for Approximations of Gradient Descent Trajectories Around Saddle Points