Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

05/03/2020
by   Bokun Wang, et al.
13

Riemannian optimization has drawn a lot of attention due to its wide applications in practice. Riemannian stochastic first-order algorithms have been studied in the literature to solve large-scale machine learning problems over Riemannian manifolds. However, most of the existing Riemannian stochastic algorithms require the objective function to be differentiable, and they do not apply to the case where the objective function is nonsmooth. In this paper, we present two Riemannian stochastic proximal gradient methods for minimizing nonsmooth function over the Stiefel manifold. The two methods, named R-ProxSGD and R-ProxSPB, are generalizations of proximal SGD and proximal SpiderBoost in Euclidean setting to the Riemannian setting. Analysis on the incremental first-order oracle (IFO) complexity of the proposed algorithms is provided. Specifically, the R-ProxSPB algorithm finds an ϵ-stationary point with O(ϵ^-3) IFOs in the online case, and O(n+√(n)ϵ^-3) IFOs in the finite-sum case with n being the number of summands in the objective. Experimental results on online sparse PCA and robust low-rank matrix completion show that our proposed methods significantly outperform the existing methods that uses Riemannian subgradient information.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/12/2019

Nonsmooth Optimization over Stiefel Manifold: Riemannian Subgradient Methods

Nonsmooth Riemannian optimization is a still under explored subfield of ...
03/25/2020

Zeroth-order Optimization on Riemannian Manifolds

We propose and analyze zeroth-order algorithms for optimization over Rie...
07/15/2022

Riemannian Natural Gradient Methods

This paper studies large-scale optimization problems on Riemannian manif...
03/13/2016

An efficient Exact-PGA algorithm for constant curvature manifolds

Manifold-valued datasets are widely encountered in many computer vision ...
11/04/2021

A Riemannian Accelerated Proximal Extragradient Framework and its Implications

The study of accelerated gradient methods in Riemannian optimization has...
10/05/2017

Primal-Dual Optimization Algorithms over Riemannian Manifolds: an Iteration Complexity Analysis

In this paper we study nonconvex and nonsmooth multi-block optimization ...
05/03/2022

Efficient implementation of incremental proximal-point methods

Model training algorithms which observe a small portion of the training ...