Benign overfitting without concentration

01/04/2021
by   Zong Shang, et al.
0

We obtain a sufficient condition for benign overfitting of linear regression problem. Our result does not rely on concentration argument but on small-ball assumption and thus can holds in heavy-tailed case. The basic idea is to establish a coordinate small-ball estimate in terms of effective rank so that we can calibrate the balance of epsilon-Net and exponential probability. Our result indicates that benign overfitting is not depending on concentration property of the input vector. Finally, we discuss potential difficulties for benign overfitting beyond linear model and a benign overfitting result without truncated effective rank.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/01/2014

Learning without Concentration

We obtain sharp bounds on the performance of Empirical Risk Minimization...
research
05/09/2023

Testing for Overfitting

High complexity models are notorious in machine learning for overfitting...
research
08/25/2021

Heavy-tailed Streaming Statistical Estimation

We consider the task of heavy-tailed statistical estimation given stream...
research
02/25/2021

Distribution-Free Robust Linear Regression

We study random design linear regression with no assumptions on the dist...
research
11/10/2021

Matrix anti-concentration inequalities with applications

We provide a polynomial lower bound on the minimum singular value of an ...
research
09/04/2017

Extending the small-ball method

The small-ball method was introduced as a way of obtaining a high probab...
research
11/18/2020

Benign Overfitting in Binary Classification of Gaussian Mixtures

Deep neural networks generalize well despite being exceedingly overparam...

Please sign up or login with your details

Forgot password? Click here to reset