On the exact computation of linear frequency principle dynamics and its generalization

10/15/2020
by   Tao Luo, et al.
7

Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that deep neural networks (DNNs) fit the target function from low to high frequency during the training, which provides insight into the training and generalization behavior of DNNs in complex tasks. In this paper, through analysis of an infinite-width two-layer NN in the neural tangent kernel (NTK) regime, we derive the exact differential equation, namely Linear Frequency-Principle (LFP) model, governing the evolution of NN output function in the frequency domain during the training. Our exact computation applies for general activation functions with no assumption on size and distribution of training data. This LFP model unravels that higher frequencies evolve polynomially or exponentially slower than lower frequencies depending on the smoothness/regularity of the activation function. We further bridge the gap between training dynamics and generalization by proving that LFP model implicitly minimizes a Frequency-Principle norm (FP-norm) of the learned function, by which higher frequencies are more severely penalized depending on the inverse of their evolution rate. Finally, we derive an a priori generalization error bound controlled by the FP-norm of the target function, which provides a theoretical justification for the empirical results that DNNs often generalize well for low frequency functions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/24/2019

Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks

It remains a puzzle that why deep neural networks (DNNs), with more para...
research
01/30/2021

Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

Why heavily parameterized neural networks (NNs) do not overfit the data ...
research
06/21/2019

Theory of the Frequency Principle for General Deep Neural Networks

Along with fruitful applications of Deep Neural Networks (DNNs) to reali...
research
07/03/2018

Training behavior of deep neural network in frequency domain

Why deep neural networks (DNNs) capable of overfitting often generalize ...
research
01/19/2019

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

We study the training process of Deep Neural Networks (DNNs) from the Fo...
research
04/03/2023

Properties and Potential Applications of Random Functional-Linked Types of Neural Networks

Random functional-linked types of neural networks (RFLNNs), e.g., the ex...
research
11/21/2022

Linear Stability Hypothesis and Rank Stratification for Nonlinear Models

Models with nonlinear architectures/parameterizations such as deep neura...

Please sign up or login with your details

Forgot password? Click here to reset