An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network

by   Tao Luo, et al.

Deep neural network (DNN) usually learns the target function from low to high frequency, which is called frequency principle or spectral bias. This frequency principle sheds light on a high-frequency curse of DNNs – difficult to learn high-frequency information. Inspired by the frequency principle, a series of works are devoted to develop algorithms for overcoming the high-frequency curse. A natural question arises: what is the upper limit of the decaying rate w.r.t. frequency when one trains a DNN? In this work, our theory, confirmed by numerical experiments, suggests that there is a critical decaying rate w.r.t. frequency in DNN training. Below the upper limit of the decaying rate, the DNN interpolates the training data by a function with a certain regularity. However, above the upper limit, the DNN interpolates the training data by a trivial function, i.e., a function is only non-zero at training data points. Our results indicate a better way to overcome the high-frequency curse is to design a proper pre-condition approach to shift high-frequency information to low-frequency one, which coincides with several previous developed algorithms for fast learning high-frequency information. More importantly, this work rigorously proves that the high-frequency curse is an intrinsic difficulty of DNNs.


page 1

page 2

page 3

page 4


A Phase Shift Deep Neural Network for High Frequency Wave Equations in Inhomogeneous Media

In this paper, we propose a phase shift deep neural network (PhaseDNN) w...

Parallel frequency function-deep neural network for efficient complex broadband signal approximation

A neural network is essentially a high-dimensional complex mapping model...

Recovering Geometric Information with Learned Texture Perturbations

Regularization is used to avoid overfitting when training a neural netwo...

Deep Learning-Based Detection of the Acute Respiratory Distress Syndrome: What Are the Models Learning?

The acute respiratory distress syndrome (ARDS) is a severe form of hypox...

Progressive transfer learning for low frequency data prediction in full waveform inversion

For the purpose of effective suppression of the cycle-skipping phenomeno...

A Quadrature Perspective on Frequency Bias in Neural Network Training with Nonuniform Data

Small generalization errors of over-parameterized neural networks (NNs) ...

PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting

Solving for the frequency-domain scattered wavefield via physics-informe...