DeepAI AI Chat
Log In Sign Up

Accelerating Neural ODEs with Spectral Elements

06/17/2019
by   Alessio Quaglino, et al.
NNAISENSE
2

This paper proposes the use of spectral element methods canuto_spectral_1988 for fast and accurate training of Neural Ordinary Differential Equations (ODE-Nets; Chen2018NeuralOD). This is achieved by expressing their dynamics as truncated series of Legendre polynomials. The series coefficients, as well as the network weights, are computed by minimizing the weighted sum of the loss function and the violation of the ODE-Net dynamics. The problem is solved by coordinate descent that alternately minimizes, with respect to the coefficients and the weights, two unconstrained sub-problems using standard backpropagation and gradient methods. The resulting optimization scheme is fully time-parallel and results in a low memory footprint. Experimental comparison to standard methods, such as backpropagation through explicit solvers and the adjoint technique Chen2018NeuralOD, on training surrogate models of small and medium-scale dynamical systems shows that it is at least one order of magnitude faster at reaching a comparable value of the loss function. The corresponding testing MSE is one order of magnitude smaller as well, suggesting generalization capabilities increase.

READ FULL TEXT

page 5

page 15

12/19/2019

Polynomial Neural Networks and Taylor maps for Dynamical Systems Simulation and Learning

The connection of Taylor maps and polynomial neural networks (PNN) to so...
07/10/2020

Learning Unstable Dynamical Systems with Time-Weighted Logarithmic Loss

When training the parameters of a linear dynamical model, the gradient d...
05/11/2020

Revealing hidden dynamics from time-series data by ODENet

To understand the hidden physical concepts from observed data is the mos...
03/22/2020

Generalization of partitioned Runge–Kutta methods for adjoint systems

This study computes the gradient of a function of numerical solutions of...
02/28/2022

Neural Ordinary Differential Equations for Nonlinear System Identification

Neural ordinary differential equations (NODE) have been recently propose...
02/19/2021

Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory

A neural network model of a differential equation, namely neural ODE, ha...