Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

06/05/2019
by   Xuanqing Liu, et al.
9

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/30/2022

Learning Robust Kernel Ensembles with Kernel Average Pooling

Model ensembles have long been used in machine learning to reduce the va...
research
11/22/2018

Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack

Recent development in the field of Deep Learning have exposed the underl...
research
10/27/2022

Noise Injection Node Regularization for Robust Learning

We introduce Noise Injection Node Regularization (NINR), a method of inj...
research
12/01/2018

Stochastic Training of Residual Networks: a Differential Equation Viewpoint

During the last few years, significant attention has been paid to the st...
research
02/15/2021

On the Inherent Regularization Effects of Noise Injection During Training

Randomly perturbing networks during the training process is a commonly u...
research
02/18/2022

Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automunge

Injecting gaussian noise into training features is well known to have re...
research
07/12/2023

Learning Stochastic Dynamical Systems as an Implicit Regularization with Graph Neural Networks

Stochastic Gumbel graph networks are proposed to learn high-dimensional ...

Please sign up or login with your details

Forgot password? Click here to reset