Neural Ordinary Differential Equations

06/19/2018
by   Tian Qi Chen, et al.
UNIVERSITY OF TORONTO
11

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a blackbox differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

READ FULL TEXT

page 5

page 6

09/18/2022

Semantic Segmentation using Neural Ordinary Differential Equations

The idea of neural Ordinary Differential Equations (ODE) is to approxima...
06/24/2021

Sparse Flows: Pruning Continuous-depth Models

Continuous deep learning architectures enable learning of flexible proba...
07/30/2020

When are Neural ODE Solutions Proper ODEs?

A key appeal of the recently proposed Neural Ordinary Differential Equat...
06/25/2021

Closed-form Continuous-Depth Models

Continuous-depth neural models, where the derivative of the model's hidd...
02/09/2021

MALI: A memory efficient and reverse accurate integrator for Neural ODEs

Neural ordinary differential equations (Neural ODEs) are a new family of...
05/19/2021

E(n) Equivariant Normalizing Flows for Molecule Generation in 3D

This paper introduces a generative model equivariant to Euclidean symmet...
05/05/2020

Time Dependence in Non-Autonomous Neural ODEs

Neural Ordinary Differential Equations (ODEs) are elegant reinterpretati...

Code Repositories

neural-ode

Jupyter notebook with Pytorch implementation of Neural Ordinary Differential Equations


view repo

Neural-Ordinary-Differential-Equations

Sample implementation of Neural Ordinary Differential Equations


view repo

neural-ode

Neural Ordinary Differential Equation


view repo

anode

[IJCAI'19, NeurIPS'19] Anode: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs


view repo

DAFT

Code for the NeurIPS 2019 paper: "Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning"


view repo

Please sign up or login with your details

Forgot password? Click here to reset