Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

by   Bokun Wang, et al.

Riemannian optimization has drawn a lot of attention due to its wide applications in practice. Riemannian stochastic first-order algorithms have been studied in the literature to solve large-scale machine learning problems over Riemannian manifolds. However, most of the existing Riemannian stochastic algorithms require the objective function to be differentiable, and they do not apply to the case where the objective function is nonsmooth. In this paper, we present two Riemannian stochastic proximal gradient methods for minimizing nonsmooth function over the Stiefel manifold. The two methods, named R-ProxSGD and R-ProxSPB, are generalizations of proximal SGD and proximal SpiderBoost in Euclidean setting to the Riemannian setting. Analysis on the incremental first-order oracle (IFO) complexity of the proposed algorithms is provided. Specifically, the R-ProxSPB algorithm finds an ϵ-stationary point with O(ϵ^-3) IFOs in the online case, and O(n+√(n)ϵ^-3) IFOs in the finite-sum case with n being the number of summands in the objective. Experimental results on online sparse PCA and robust low-rank matrix completion show that our proposed methods significantly outperform the existing methods that uses Riemannian subgradient information.


page 1

page 2

page 3

page 4


Nonsmooth Optimization over Stiefel Manifold: Riemannian Subgradient Methods

Nonsmooth Riemannian optimization is a still under explored subfield of ...

Zeroth-order Optimization on Riemannian Manifolds

We propose and analyze zeroth-order algorithms for optimization over Rie...

Riemannian Natural Gradient Methods

This paper studies large-scale optimization problems on Riemannian manif...

An efficient Exact-PGA algorithm for constant curvature manifolds

Manifold-valued datasets are widely encountered in many computer vision ...

A Riemannian Accelerated Proximal Extragradient Framework and its Implications

The study of accelerated gradient methods in Riemannian optimization has...

Primal-Dual Optimization Algorithms over Riemannian Manifolds: an Iteration Complexity Analysis

In this paper we study nonconvex and nonsmooth multi-block optimization ...

Efficient implementation of incremental proximal-point methods

Model training algorithms which observe a small portion of the training ...