Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

03/16/2023
by   Sholom Schechtman, et al.
0

We consider the problem of minimizing a non-convex function over a smooth manifold ℳ. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards ℳ. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities 𝒪(1/ε^2) and 𝒪(1/ε^4) in the deterministic and stochastic cases, respectively. Furthermore, we establish that, under an appropriate choice of the projection metric, our method recovers the landing algorithm of Ablin and Peyré (2022), a recently introduced algorithm for optimization over the Stiefel manifold. As a result, we significantly extend the analysis of Ablin and Peyré (2022), establishing near-optimal rates both in deterministic and stochastic frameworks. Finally, we perform numerical experiments which shows the efficiency of ODCGM in a high-dimensional setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/30/2019

UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimization

We propose a novel adaptive, accelerated algorithm for the stochastic co...
research
08/11/2022

Near-Optimal Algorithms for Making the Gradient Small in Stochastic Minimax Optimization

We study the problem of finding a near-stationary point for smooth minim...
research
02/15/2021

Fast and accurate optimization on the orthogonal manifold without retraction

We consider the problem of minimizing a function over the manifold of or...
research
06/27/2019

Near-Optimal Methods for Minimizing Star-Convex Functions and Beyond

In this paper, we provide near-optimal accelerated first-order methods f...
research
03/23/2023

Chordal Averaging on Flag Manifolds and Its Applications

This paper presents a new, provably-convergent algorithm for computing t...
research
08/17/2020

A near-optimal stochastic gradient method for decentralized non-convex finite-sum optimization

This paper describes a near-optimal stochastic first-order gradient meth...
research
04/10/2023

First-order methods for Stochastic Variational Inequality problems with Function Constraints

The monotone Variational Inequality (VI) is an important problem in mach...

Please sign up or login with your details

Forgot password? Click here to reset