Benign Overfitting in Binary Classification of Gaussian Mixtures

11/18/2020
by   Ke Wang, et al.
0

Deep neural networks generalize well despite being exceedingly overparametrized, but understanding the statistical principles behind this so called benign-overfitting phenomenon is not yet well understood. Recently there has been remarkable progress towards understanding benign-overfitting in simpler models, such as linear regression and, even more recently, linear classification. This paper studies benign-overfitting for data generated from a popular binary Gaussian mixtures model (GMM) and classifiers trained by support-vector machines (SVM). Our approach has two steps. First, we leverage an idea introduced in (Muthukumar et al. 2020) to relate the SVM solution to the least-squares (LS) solution. Second, we derive novel non-asymptotic bounds on the classification error of LS solution. Combining the two gives sufficient conditions on the overparameterization ratio and the signal-to-noise ratio that lead to benign overfitting. We corroborate our theoretical findings with numerical simulations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/21/2021

Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation

The growing literature on "benign overfitting" in overparameterized mode...
research
02/14/2022

Benign Overfitting in Two-layer Convolutional Neural Networks

Modern neural networks often have great expressive power and can be trai...
research
04/28/2021

Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures

Modern machine learning systems such as deep neural networks are often h...
research
05/28/2021

Support vector machines and linear regression coincide with very high-dimensional features

The support vector machine (SVM) and minimum Euclidean norm least square...
research
08/08/2019

Optimal multiclass overfitting by sequence reconstruction from Hamming queries

A primary concern of excessive reuse of test datasets in machine learnin...
research
04/08/2019

Feature Learning Viewpoint of AdaBoost and a New Algorithm

The AdaBoost algorithm has the superiority of resisting overfitting. Und...
research
01/04/2021

Benign overfitting without concentration

We obtain a sufficient condition for benign overfitting of linear regres...

Please sign up or login with your details

Forgot password? Click here to reset