ANODEV2: A Coupled Neural ODE Evolution Framework

06/10/2019
by   Tianjun Zhang, et al.
3

It has been observed that residual networks can be viewed as the explicit Euler discretization of an Ordinary Differential Equation (ODE). This observation motivated the introduction of so-called Neural ODEs, which allow more general discretization schemes with adaptive time stepping. Here, we propose ANODEV2, which is an extension of this approach that also allows evolution of the neural network parameters, in a coupled ODE-based formulation. The Neural ODE method introduced earlier is in fact a special case of this new more general framework. We present the formulation of ANODEV2, derive optimality conditions, and implement a coupled reaction-diffusion-advection version of this framework in PyTorch. We present empirical results using several different configurations of ANODEV2, testing them on multiple models on CIFAR-10. We report results showing that this coupled ODE-based framework is indeed trainable, and that it achieves higher accuracy, as compared to the baseline models as well as the recently-proposed Neural ODE approach.

READ FULL TEXT

page 5

page 6

page 10

page 12

research
02/27/2019

ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs

Residual neural networks can be viewed as the forward Euler discretizati...
research
04/23/2021

Numerical Methods for the Hyperbolic Monge-Ampère Equation Based on the Method of Characteristics

We present three alternative derivations of the method of characteristic...
research
04/06/2021

ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation

It has been found that residual networks are an Euler discretization of ...
research
07/24/2019

Estimation of ordinary differential equation models with discretization error quantification

We consider estimation of ordinary differential equation (ODE) models fr...
research
11/17/2020

On mathematical aspects of evolution of dislocation density in metallic materials

This paper deals with the solution of delay differential equations descr...
research
05/19/2021

Coupled-Cluster Theory Revisited

We propose a comprehensive mathematical framework for Coupled-Cluster-ty...

Please sign up or login with your details

Forgot password? Click here to reset