DeepAI AI Chat
Log In Sign Up

Generalization in Supervised Learning Through Riemannian Contraction

by   Leo Kozachkov, et al.
University of Notre Dame

We prove that Riemannian contraction in a supervised learning setting implies generalization. Specifically, we show that if an optimizer is contracting in some Riemannian metric with rate λ > 0, it is uniformly algorithmically stable with rate 𝒪(1/λ n), where n is the number of labelled examples in the training set. The results hold for stochastic and deterministic optimization, in both continuous and discrete-time, for convex and non-convex loss surfaces. The associated generalization bounds reduce to well-known results in the particular case of gradient descent over convex or strongly convex loss surfaces. They can be shown to be optimal in certain linear settings, such as kernel ridge regression under gradient flow.


page 2

page 4


Variance reduction for Riemannian non-convex optimization with batch size adaptation

Variance reduction techniques are popular in accelerating gradient desce...

Stochastic Variance Reduced Riemannian Eigensolver

We study the stochastic Riemannian gradient algorithm for matrix eigen-d...

Averaging Stochastic Gradient Descent on Riemannian Manifolds

We consider the minimization of a function defined on a Riemannian manif...

First-Order Algorithms for Min-Max Optimization in Geodesic Metric Spaces

From optimal transport to robust dimensionality reduction, a plethora of...

Computation and verification of contraction metrics for exponentially stable equilibria

The determination of exponentially stable equilibria and their basin of ...

Robust Controller Design for Stochastic Nonlinear Systems via Convex Optimization

This paper presents ConVex optimization-based Stochastic steady-state Tr...