DeepAI AI Chat
Log In Sign Up

Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints

by   Pierre Ablin, et al.
Chinese Academy of Science
Apple Inc.
Université catholique de Louvain

Orthogonality constraints naturally appear in many machine learning problems, from Principal Components Analysis to robust neural network training. They are usually solved using Riemannian optimization algorithms, which minimize the objective function while enforcing the constraint. However, enforcing the orthogonality constraint can be the most time-consuming operation in such algorithms. Recently, Ablin Peyré (2022) proposed the Landing algorithm, a method with cheap iterations that does not enforce the orthogonality constraint but is attracted towards the manifold in a smooth manner. In this article, we provide new practical and theoretical developments for the landing algorithm. First, the method is extended to the Stiefel manifold, the set of rectangular orthogonal matrices. We also consider stochastic and variance reduction algorithms when the cost function is an average of many functions. We demonstrate that all these methods have the same rate of convergence as their Riemannian counterparts that exactly enforce the constraint. Finally, our experiments demonstrate the promise of our approach to an array of machine-learning problems that involve orthogonality constraints.


page 1

page 2

page 3

page 4


Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Riemannian optimization has drawn a lot of attention due to its wide app...

Riemannian stochastic variance reduced gradient on Grassmann manifold

Stochastic variance reduction algorithms have recently become popular fo...

Decentralized Riemannian Algorithm for Nonconvex Minimax Problems

The minimax optimization over Riemannian manifolds (possibly nonconvex c...

Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform

Strictly enforcing orthonormality constraints on parameter matrices has ...

Manifold Free Riemannian Optimization

Riemannian optimization is a principled framework for solving optimizati...

Riemannian Stochastic Approximation for Minimizing Tame Nonsmooth Objective Functions

In many learning applications, the parameters in a model are structurall...

Multi-Rank Sparse and Functional PCA: Manifold Optimization and Iterative Deflation Techniques

We consider the problem of estimating multiple principal components usin...