Escape saddle points faster on manifolds via perturbed Riemannian stochastic recursive gradient

10/23/2020
by   Andi Han, et al.
1

In this paper, we propose a variant of Riemannian stochastic recursive gradient method that can achieve second-order convergence guarantee using simple perturbation. The idea is to perturb the iterates when gradient is small and carry out stochastic recursive gradient updates over tangent space. This avoids the complication of exploiting Riemannian geometry. We show that under finite-sum setting, our algorithm requires 𝒪( √(n)/ϵ^2 + √(n)/δ^4 + n/δ^3) stochastic gradient queries to find a (ϵ, δ)-second-order critical point. This strictly improves the complexity of perturbed Riemannian gradient descent and is superior to perturbed Riemannian accelerated gradient descent under large-sample settings. We also provide a complexity of 𝒪(1/ϵ^3 + 1/δ^3 ϵ^2 + 1/δ^4 ϵ) for online optimization, which is novel on Riemannian manifold in terms of second-order convergence using only first-order information.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/10/2019

Efficiently escaping saddle points on manifolds

Smooth, non-convex optimization problems on Riemannian manifolds occur i...
research
07/31/2017

Statistics on the (compact) Stiefel manifold: Theory and Applications

A Stiefel manifold of the compact type is often encountered in many fiel...
research
08/05/2020

An accelerated first-order method for non-convex optimization on manifolds

We describe the first gradient methods on Riemannian manifolds to achiev...
research
09/30/2021

Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization

We show that stochastic acceleration can be achieved under the perturbed...
research
04/25/2022

Accelerated Multiplicative Weights Update Avoids Saddle Points almost always

We consider non-convex optimization problems with constraint that is a p...
research
02/01/2023

Riemannian Stochastic Approximation for Minimizing Tame Nonsmooth Objective Functions

In many learning applications, the parameters in a model are structurall...

Please sign up or login with your details

Forgot password? Click here to reset