DeepAI AI Chat
Log In Sign Up

A neuron-wise subspace correction method for the finite neuron method

by   Jongho Park, et al.
King Abdullah University of Science and Technology
KAIST 수리과학과
Penn State University

In this paper, we propose a novel algorithm called Neuron-wise Parallel Subspace Correction Method (NPSC) for training ReLU neural networks for numerical solution of partial differential equations (PDEs). Despite of extremely extensive research activities in applying neural networks for numerical PDEs, there is still a serious lack of training algorithms that can be used to obtain approximation with adequate accuracy. Based on recent results on the spectral properties of linear layers and landscape analysis for single neuron problems, we develop a special type of subspace correction method that deals with the linear layer and each neuron in the nonlinear layer separately. An optimal preconditioner that resolves the ill-conditioning of the linear layer is presented, so that the linear layer is trained in a uniform number of iterations with respect to the number of neurons. In each single neuron problem, a good local minimum is found by a superlinearly convergent algorithm, avoiding regions where the loss function is flat. Performance of the proposed method is demonstrated through numerical experiments for function approximation problems and PDEs.


page 1

page 2

page 3

page 4


Adaptive Two-Layer ReLU Neural Network: II. Ritz Approximation to Elliptic PDEs

In this paper, we study adaptive neuron enhancement (ANE) method for sol...

Neural networks-based backward scheme for fully nonlinear PDEs

We propose a numerical method for solving high dimensional fully nonline...

Adaptive Two-Layer ReLU Neural Network: I. Best Least-squares Approximation

In this paper, we introduce adaptive neuron enhancement (ANE) method for...

The effect of time discretization on the solution of parabolic PDEs with ANNs

We investigate the resolution of parabolic PDEs via Extreme Learning Mac...

Deep Ritz revisited

Recently, progress has been made in the application of neural networks t...