AI Chat AI Image Generator AI Video Text to Speech

Noisy Interpolation Learning with Shallow Univariate ReLU Networks

07/28/2023

∙

by Nirmit Joshi, et al.

∙

∙

We study the asymptotic overfitting behavior of interpolation with minimum norm (ℓ_2 of the weights) two-layer ReLU networks for noisy univariate regression. We show that overfitting is tempered for the L_1 loss, and any L_p loss for p<2, but catastrophic for p≥ 2.

Nirmit Joshi
3 publications
Gal Vardi
25 publications
Nathan Srebro
96 publications

page 1

page 2

page 3

page 4

research

∙ 11/10/2021

Tight bounds for minimum l1-norm interpolation of noisy data

We provide matching upper and lower bounds of order σ^2/log(d/n) for the...

0 Guillaume Wang, et al. ∙

research

∙ 02/14/2023

Interpolation Learning With Minimum Description Length

We prove that the Minimum Description Length learning rule exhibits temp...

0 Naren Sarayu Manoj, et al. ∙

research

∙ 06/18/2019

Gradient Dynamics of Shallow Univariate ReLU Networks

We present a theoretical and empirical study of the gradient dynamics of...

3 Francis Williams, et al. ∙

research

∙ 06/16/2023

Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?

We study benign overfitting in two-layer ReLU networks trained using gra...

0 Erin George, et al. ∙

research

∙ 09/27/2021

Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions

We prove a precise geometric description of all one layer ReLU networks ...

0 Boris Hanin, et al. ∙

research

∙ 03/02/2023

Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization

Linear classifiers and leaky ReLU networks trained by gradient flow on t...

0 Spencer Frei, et al. ∙

research

∙ 05/24/2023

From Tempered to Benign Overfitting in ReLU Neural Networks

Overparameterized neural networks (NNs) are observed to generalize well ...

0 Guy Kornowski, et al. ∙