Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks

06/29/2022
by   Ghazal Farhani, et al.
0

Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, they often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why training a model via the Adam optimizer can accelerate the convergence while reducing the spectral bias. Moreover, our numerical experiments have confirmed that wide-enough networks using SGDM still converge to desirable solutions, even in the presence of high-frequency features. In fact, we show that the width of a network plays a critical role in convergence.

READ FULL TEXT

page 7

page 8

page 9

research
11/12/2022

Pre-training strategy for solving evolution equations based on physics-informed neural networks

The physics informed neural network (PINN) is a promising method for sol...
research
12/02/2021

Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks

The Neural network-based approach to solving partial differential equati...
research
06/22/2023

Physical informed neural networks with soft and hard boundary constraints for solving advection-diffusion equations using Fourier expansions

Deep learning methods have gained considerable interest in the numerical...
research
03/03/2023

Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks

Physics-informed neural networks (PINNs) have effectively been demonstra...
research
01/07/2023

Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders

An important inference from Neural Tangent Kernel (NTK) theory is the ex...
research
09/20/2021

Learning in Sinusoidal Spaces with Physics-Informed Neural Networks

A physics-informed neural network (PINN) uses physics-augmented loss fun...
research
09/29/2021

PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting

Solving for the frequency-domain scattered wavefield via physics-informe...

Please sign up or login with your details

Forgot password? Click here to reset