Harmless interpolation in regression and classification with structured features

11/09/2021
by   Andrew D. McRae, et al.
0

Overparametrized neural networks tend to perfectly fit noisy training data yet generalize well on test data. Inspired by this empirical observation, recent work has sought to understand this phenomenon of benign overfitting or harmless interpolation in the much simpler linear model. Previous theoretical work critically assumes that either the data features are statistically independent or the input data is high-dimensional; this precludes general nonparametric settings with structured feature maps. In this paper, we present a general and flexible framework for upper bounding regression and classification risk in a reproducing kernel Hilbert space. A key contribution is that our framework describes precise sufficient conditions on the data Gram matrix under which harmless interpolation occurs. Our results recover prior independent-features results (with a much simpler analysis), but they furthermore show that harmless interpolation can occur in more general settings such as features that are a bounded orthonormal system. Furthermore, our results show an asymptotic separation between classification and regression performance in a manner that was previously only shown for Gaussian features.

READ FULL TEXT

Authors

page 1

page 2

page 3

page 4

12/28/2018

Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenon

We show that minimum-norm interpolation in the Reproducing Kernel Hilber...
06/07/2021

Parameter-free Statistically Consistent Interpolation: Dimension-independent Convergence Rates for Hilbert kernel regression

Previously, statistical textbook wisdom has held that interpolating nois...
08/01/2018

Just Interpolate: Kernel "Ridgeless" Regression Can Generalize

In the absence of explicit regularization, Kernel "Ridgeless" Regression...
03/19/2019

Surprises in High-Dimensional Ridgeless Least Squares Interpolation

Interpolators -- estimators that achieve zero training error -- have att...
12/08/2021

Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression

We study a localized notion of uniform convergence known as an "optimist...
06/25/2018

Does data interpolation contradict statistical optimality?

We show that learning methods interpolating the training data can achiev...
05/28/2019

NIL: Learning Nonlinear Interpolants

Nonlinear interpolants have been shown useful for the verification of pr...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.