Delay Differential Neural Networks

12/12/2020
by   Srinivas Anumasa, et al.
0

Neural ordinary differential equations (NODEs) treat computation of intermediate feature vectors as trajectories of ordinary differential equation parameterized by a neural network. In this paper, we propose a novel model, delay differential neural networks (DDNN), inspired by delay differential equations (DDEs). The proposed model considers the derivative of the hidden feature vector as a function of the current feature vector and past feature vectors (history). The function is modelled as a neural network and consequently, it leads to continuous depth alternatives to many recent ResNet variants. We propose two different DDNN architectures, depending on the way current and past feature vectors are considered. For training DDNNs, we provide a memory-efficient adjoint method for computing gradients and back-propagate through the network. DDNN improves the data efficiency of NODE by further reducing the number of parameters without affecting the generalization performance. Experiments conducted on synthetic and real-world image classification datasets such as Cifar10 and Cifar100 show the effectiveness of the proposed models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/28/2022

Learning Time Delay Systems with Neural Ordinary Differential Equations

A novel way of using neural networks to learn the dynamics of time delay...
research
02/22/2021

Neural Delay Differential Equations

Neural Ordinary Differential Equations (NODEs), a framework of continuou...
research
04/03/2023

Learning the Delay Using Neural Delay Differential Equations

The intersection of machine learning and dynamical systems has generated...
research
09/18/2022

Semantic Segmentation using Neural Ordinary Differential Equations

The idea of neural Ordinary Differential Equations (ODE) is to approxima...
research
05/05/2020

Time Dependence in Non-Autonomous Neural ODEs

Neural Ordinary Differential Equations (ODEs) are elegant reinterpretati...
research
07/04/2021

Learning ODEs via Diffeomorphisms for Fast and Robust Integration

Advances in differentiable numerical integrators have enabled the use of...
research
06/10/2020

Interpolation between Residual and Non-Residual Networks

Although ordinary differential equations (ODEs) provide insights for des...

Please sign up or login with your details

Forgot password? Click here to reset