DeepAI AI Chat
Log In Sign Up

Gradient Methods Never Overfit On Separable Data

by   Ohad Shamir, et al.

A line of recent works established that when training linear predictors over separable data, using gradient methods and exponentially-tailed losses, the predictors asymptotically converge in direction to the max-margin predictor. As a consequence, the predictors asymptotically do not overfit. However, this does not address the question of whether overfitting might occur non-asymptotically, after some bounded number of iterations. In this paper, we formally show that standard gradient methods (in particular, gradient flow, gradient descent and stochastic gradient descent) never overfit on separable data: If we run these methods for T iterations on a dataset of size m, both the empirical risk and the generalization error decrease at an essentially optimal rate of 𝒪̃(1/γ^2 T) up till T≈ m, at which point the generalization error remains fixed at an essentially optimal level of 𝒪̃(1/γ^2 m) regardless of how large T is. Along the way, we present non-asymptotic bounds on the number of margin violations over the dataset, and prove their tightness.


page 1

page 2

page 3

page 4

∙ 02/27/2022

Stability vs Implicit Bias of Gradient Methods on Separable Data and Beyond

An influential line of recent work has focused on the generalization pro...
∙ 06/19/2020

Gradient descent follows the regularization path for general losses

Recent work across many machine learning disciplines has highlighted tha...
∙ 05/22/2023

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data

Normalized gradient descent has shown substantial success in speeding up...
∙ 05/22/2019

Convergence and Margin of Adversarial Training on Separable Data

Adversarial training is a technique for training robust machine learning...
∙ 09/15/2022

Decentralized Learning with Separable Data: Generalization and Fast Algorithms

Decentralized learning offers privacy and communication efficiency when ...
∙ 01/27/2022

The Implicit Bias of Benign Overfitting

The phenomenon of benign overfitting, where a predictor perfectly fits n...
∙ 12/08/2018

Weighted Risk Minimization & Deep Learning

Importance weighting is a key ingredient in many algorithms for causal i...