DeepAI AI Chat
Log In Sign Up

Generalization in Supervised Learning Through Riemannian Contraction

01/17/2022
by   Leo Kozachkov, et al.
MIT
University of Notre Dame
0

We prove that Riemannian contraction in a supervised learning setting implies generalization. Specifically, we show that if an optimizer is contracting in some Riemannian metric with rate λ > 0, it is uniformly algorithmically stable with rate 𝒪(1/λ n), where n is the number of labelled examples in the training set. The results hold for stochastic and deterministic optimization, in both continuous and discrete-time, for convex and non-convex loss surfaces. The associated generalization bounds reduce to well-known results in the particular case of gradient descent over convex or strongly convex loss surfaces. They can be shown to be optimal in certain linear settings, such as kernel ridge regression under gradient flow.

READ FULL TEXT

page 2

page 4

07/03/2020

Variance reduction for Riemannian non-convex optimization with batch size adaptation

Variance reduction techniques are popular in accelerating gradient desce...
05/26/2016

Stochastic Variance Reduced Riemannian Eigensolver

We study the stochastic Riemannian gradient algorithm for matrix eigen-d...
02/26/2018

Averaging Stochastic Gradient Descent on Riemannian Manifolds

We consider the minimization of a function defined on a Riemannian manif...
06/04/2022

First-Order Algorithms for Min-Max Optimization in Geodesic Metric Spaces

From optimal transport to robust dimensionality reduction, a plethora of...
09/20/2019

Computation and verification of contraction metrics for exponentially stable equilibria

The determination of exponentially stable equilibria and their basin of ...
07/10/2020

Robust Controller Design for Stochastic Nonlinear Systems via Convex Optimization

This paper presents ConVex optimization-based Stochastic steady-state Tr...