An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network

05/25/2021
by   Tao Luo, et al.
4

Deep neural network (DNN) usually learns the target function from low to high frequency, which is called frequency principle or spectral bias. This frequency principle sheds light on a high-frequency curse of DNNs – difficult to learn high-frequency information. Inspired by the frequency principle, a series of works are devoted to develop algorithms for overcoming the high-frequency curse. A natural question arises: what is the upper limit of the decaying rate w.r.t. frequency when one trains a DNN? In this work, our theory, confirmed by numerical experiments, suggests that there is a critical decaying rate w.r.t. frequency in DNN training. Below the upper limit of the decaying rate, the DNN interpolates the training data by a function with a certain regularity. However, above the upper limit, the DNN interpolates the training data by a trivial function, i.e., a function is only non-zero at training data points. Our results indicate a better way to overcome the high-frequency curse is to design a proper pre-condition approach to shift high-frequency information to low-frequency one, which coincides with several previous developed algorithms for fast learning high-frequency information. More importantly, this work rigorously proves that the high-frequency curse is an intrinsic difficulty of DNNs.

READ FULL TEXT

page 1

page 2

page 3

page 4

09/23/2019

A Phase Shift Deep Neural Network for High Frequency Wave Equations in Inhomogeneous Media

In this paper, we propose a phase shift deep neural network (PhaseDNN) w...
06/19/2021

Parallel frequency function-deep neural network for efficient complex broadband signal approximation

A neural network is essentially a high-dimensional complex mapping model...
01/20/2020

Recovering Geometric Information with Learned Texture Perturbations

Regularization is used to avoid overfitting when training a neural netwo...
09/25/2021

Deep Learning-Based Detection of the Acute Respiratory Distress Syndrome: What Are the Models Learning?

The acute respiratory distress syndrome (ARDS) is a severe form of hypox...
12/20/2019

Progressive transfer learning for low frequency data prediction in full waveform inversion

For the purpose of effective suppression of the cycle-skipping phenomeno...
05/28/2022

A Quadrature Perspective on Frequency Bias in Neural Network Training with Nonuniform Data

Small generalization errors of over-parameterized neural networks (NNs) ...
09/29/2021

PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting

Solving for the frequency-domain scattered wavefield via physics-informe...