Self-adaptive deep neural network: Numerical approximation to functions and PDEs

by   Zhiqiang Cai, et al.

Designing an optimal deep neural network for a given task is important and challenging in many machine learning applications. To address this issue, we introduce a self-adaptive algorithm: the adaptive network enhancement (ANE) method, written as loops of the form train, estimate and enhance. Starting with a small two-layer neural network (NN), the step train is to solve the optimization problem at the current NN; the step estimate is to compute a posteriori estimator/indicators using the solution at the current NN; the step enhance is to add new neurons to the current NN. Novel network enhancement strategies based on the computed estimator/indicators are developed in this paper to determine how many new neurons and when a new layer should be added to the current NN. The ANE method provides a natural process for obtaining a good initialization in training the current NN; in addition, we introduce an advanced procedure on how to initialize newly added neurons for a better approximation. We demonstrate that the ANE method can automatically design a nearly minimal NN for learning functions exhibiting sharp transitional layers as well as discontinuous solutions of hyperbolic partial differential equations.


page 1

page 2

page 3

page 4


Adaptive Two-Layer ReLU Neural Network: II. Ritz Approximation to Elliptic PDEs

In this paper, we study adaptive neuron enhancement (ANE) method for sol...

Adaptive Two-Layer ReLU Neural Network: I. Best Least-squares Approximation

In this paper, we introduce adaptive neuron enhancement (ANE) method for...

The estimation of training accuracy for two-layer neural networks on random datasets without training

Although the neural network (NN) technique plays an important role in ma...

Understanding Self-attention Mechanism via Dynamical System Perspective

The self-attention mechanism (SAM) is widely used in various fields of a...

Persistent Neurons

Most algorithms used in neural networks(NN)-based leaning tasks are stro...

A Cascade Neural Network Architecture investigating Surface Plasmon Polaritons propagation for thin metals in OpenMP

Surface plasmon polaritons (SPPs) confined along metal-dielectric interf...

A Neural Network-Based Enrichment of Reproducing Kernel Approximation for Modeling Brittle Fracture

Numerical modeling of localizations is a challenging task due to the evo...

Please sign up or login with your details

Forgot password? Click here to reset