Meta-Solver for Neural Ordinary Differential Equations

03/15/2021
by   Julia Gusak, et al.
0

A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization method and its properties, which may not be optimal for the selected application and yield the overfitting to the given solver. In our paper, we investigate how the variability in solvers' space can improve neural ODEs performance. We consider a family of Runge-Kutta methods that are parameterized by no more than two scalar variables. Based on the solvers' properties, we propose an approach to decrease neural ODEs overfitting to the pre-defined solver, along with a criterion to evaluate such behaviour. Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks. Recently it was shown that neural ODEs demonstrate superiority over conventional CNNs in terms of robustness. Our work demonstrates that the model robustness can be further improved by optimizing solver choice for a given task. The source code to reproduce our experiments is available at https://github.com/juliagusak/neural-ode-metasolver.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/05/2022

LyaNet: A Lyapunov Framework for Training Neural ODEs

We propose a method for training ordinary differential equations by usin...
research
06/27/2022

Zero Stability Well Predicts Performance of Convolutional Neural Networks

The question of what kind of convolutional neural network (CNN) structur...
research
11/13/2022

Experimental study of Neural ODE training with adaptive solver for dynamical systems modeling

Neural Ordinary Differential Equations (ODEs) was recently introduced as...
research
04/07/2020

Offsite Autotuning Approach – Performance Model Driven Autotuning Applied to Parallel Explicit ODE Methods

Autotuning techniques are a promising approach to minimize the otherwise...
research
05/16/2023

Ortho-ODE: Enhancing Robustness and of Neural ODEs against Adversarial Attacks

Neural Ordinary Differential Equations (NODEs) probed the usage of numer...
research
03/09/2023

Efficient Certified Training and Robustness Verification of Neural ODEs

Neural Ordinary Differential Equations (NODEs) are a novel neural archit...
research
10/10/2021

Heavy Ball Neural Ordinary Differential Equations

We propose heavy ball neural ordinary differential equations (HBNODEs), ...

Please sign up or login with your details

Forgot password? Click here to reset