
Generalization Bounds for Uniformly Stable Algorithms
Uniform stability of a learning algorithm is a classical notion of algor...
read it

Stability and Deviation Optimal Risk Bounds with Convergence Rate O(1/n)
The sharpest known high probability generalization bounds for uniformly ...
read it

Sharper bounds for uniformly stable algorithms
The generalization bounds for stable algorithms is a classical question ...
read it

Finegrained Generalization Analysis of Structured Output Prediction
In machine learning we often encounter structured output prediction prob...
read it

Stability and Generalization of Stochastic Gradient Methods for Minimax Problems
Many machine learning problems can be formulated as minimax problems suc...
read it

A Note on HighProbability versus InExpectation Guarantees of Generalization Bounds in Machine Learning
Statistical machine learning theory often tries to give generalization g...
read it

Convolutional Analysis Operator Learning: Dependence on Training Data
Convolutional analysis operator learning (CAOL) enables the unsupervised...
read it
High probability generalization bounds for uniformly stable algorithms with nearly optimal rate
Algorithmic stability is a classical approach to understanding and analysis of the generalization error of learning algorithms. A notable weakness of most stabilitybased generalization bounds is that they hold only in expectation. Generalization with high probability has been established in a landmark paper of Bousquet and Elisseeff (2002) albeit at the expense of an additional √(n) factor in the bound. Specifically, their bound on the estimation error of any γuniformly stable learning algorithm on n samples and range in [0,1] is O(γ√(n (1/δ)) + √((1/δ)/n)) with probability ≥ 1δ. The √(n) overhead makes the bound vacuous in the common settings where γ≥ 1/√(n). A stronger bound was recently proved by the authors (Feldman and Vondrak, 2018) that reduces the overhead to at most O(n^1/4). Still, both of these results give optimal generalization bounds only when γ = O(1/n). We prove a nearly tight bound of O(γ(n)(n/δ) + √((1/δ)/n)) on the estimation error of any γuniformly stable algorithm. It implies that algorithms that are uniformly stable with γ = O(1/√(n)) have essentially the same estimation error as algorithms that output a fixed function. Our result leads to the first highprobability generalization bounds for multipass stochastic gradient descent and regularized ERM for stochastic convex problems with nearly optimal rate  resolving open problems in prior work. Our proof technique is new and we introduce several analysis tools that might find additional applications.
READ FULL TEXT
Comments
There are no comments yet.