Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network

01/06/2022
by   Huaiqian You, et al.
44

Neural operators have recently become popular tools for designing solution maps between function spaces in the form of neural networks. Differently from classical scientific machine learning approaches that learn parameters of a known partial differential equation (PDE) for a single instance of the input parameters at a fixed resolution, neural operators approximate the solution map of a family of PDEs. Despite their success, the uses of neural operators are so far restricted to relatively shallow neural networks and confined to learning hidden governing laws. In this work, we propose a novel nonlocal neural operator, which we refer to as nonlocal kernel network (NKN), that is resolution independent, characterized by deep neural networks, and capable of handling a variety of tasks such as learning governing equations and classifying images. Our NKN stems from the interpretation of the neural network as a discrete nonlocal diffusion reaction equation that, in the limit of infinite layers, is equivalent to a parabolic nonlocal equation, whose stability is analyzed via nonlocal vector calculus. The resemblance with integral forms of neural operators allows NKNs to capture long-range dependencies in the feature space, while the continuous treatment of node-to-node interactions makes NKNs resolution independent. The resemblance with neural ODEs, reinterpreted in a nonlocal sense, and the stable network dynamics between layers allow for generalization of NKN's optimal parameters from shallow to deep networks. This fact enables the use of shallow-to-deep initialization techniques. Our tests show that NKNs outperform baseline methods in both learning governing equations and image classification tasks and generalize well to different resolutions and depths.

READ FULL TEXT

page 22

page 23

page 24

research
01/28/2022

Pseudo-Differential Integral Operator for Learning Solution Operators of Partial Differential Equations

Learning mapping between two function spaces has attracted considerable ...
research
04/23/2022

U-NO: U-shaped Neural Operators

Neural operators generalize classical neural networks to maps between in...
research
03/07/2020

Neural Operator: Graph Kernel Network for Partial Differential Equations

The classical development of neural networks has been primarily for mapp...
research
03/06/2019

IMEXnet: A Forward Stable Deep Neural Network

Deep convolutional neural networks have revolutionized many machine lear...
research
09/28/2021

Multiwavelet-based Operator Learning for Differential Equations

The solution of a partial differential equation can be obtained by compu...
research
06/18/2020

A Shooting Formulation of Deep Learning

Continuous-depth neural networks can be viewed as deep limits of discret...
research
06/06/2023

Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere

Fourier Neural Operators (FNOs) have proven to be an efficient and effec...

Please sign up or login with your details

Forgot password? Click here to reset