Overview frequency principle/spectral bias in deep learning

01/19/2022
by   Zhi-Qin John Xu, et al.
0

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights into this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) – DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by one-dimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.

READ FULL TEXT

page 7

page 10

research
07/03/2018

Training behavior of deep neural network in frequency domain

Why deep neural networks (DNNs) capable of overfitting often generalize ...
research
01/04/2021

Frequency Principle in Deep Learning Beyond Gradient-descent-based Training

Frequency perspective recently makes progress in understanding deep lear...
research
04/29/2020

Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs

Over-parameterized deep neural networks (DNNs) with sufficient capacity ...
research
01/19/2019

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

We study the training process of Deep Neural Networks (DNNs) from the Fo...
research
07/28/2020

Deep frequency principle towards understanding why deeper learning is faster

Understanding the effect of depth in deep learning is a critical problem...
research
04/03/2023

Properties and Potential Applications of Random Functional-Linked Types of Neural Networks

Random functional-linked types of neural networks (RFLNNs), e.g., the ex...
research
10/10/2022

DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learning

Causal mediation analysis can unpack the black box of causality and is t...

Please sign up or login with your details

Forgot password? Click here to reset