The Slow Deterioration of the Generalization Error of the Random Feature Model

08/13/2020 ∙ by Chao Ma, et al. ∙ 45 ∙ share

The random feature model exhibits a kind of resonance behavior when the number of parameters is close to the training sample size. This behavior is characterized by the appearance of large generalization gap, and is due to the occurrence of very small eigenvalues for the associated Gram matrix. In this paper, we examine the dynamic behavior of the gradient descent algorithm in this regime. We show, both theoretically and experimentally, that there is a dynamic self-correction mechanism at work: The larger the eventual generalization gap, the slower it develops, both because of the small eigenvalues. This gives us ample time to stop the training process and obtain solutions with good generalization property.

READ FULL TEXT

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Background: Deep learning models are typically trained using stochastic ...
05/24/2017 ∙ by Elad Hoffer, et al. ∙ 0 ∙ share

read it
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers

Neural networks have great success in many machine learning applications...
11/12/2018 ∙ by Zeyuan Allen-Zhu, et al. ∙ 0 ∙ share

read it
High-dimensional dynamics of generalization error in neural networks

We perform an average case analysis of the generalization dynamics of la...
10/10/2017 ∙ by Madhu S. Advani, et al. ∙ 0 ∙ share

read it
A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

A fairly comprehensive analysis is presented for the gradient descent dy...
04/08/2019 ∙ by Weinan E, et al. ∙ 25 ∙ share

read it
Benign Overfitting and Noisy Features

Modern machine learning often operates in the regime where the number of...
08/06/2020 ∙ by Zhu Li, et al. ∙ 21 ∙ share

read it
The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models

A numerical and phenomenological study of the gradient descent (GD) algo...
06/25/2020 ∙ by Chao Ma, et al. ∙ 6 ∙ share

read it

The Slow Deterioration of the Generalization Error of the Random Feature Model

Authors

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers

High-dimensional dynamics of generalization error in neural networks

A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

Benign Overfitting and Noisy Features

The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models

This week in AI