Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families

11/23/2019
by   Yibo Yang, et al.
0

The correspondence between residual networks and dynamical systems motivates researchers to unravel the physics of ResNets with well-developed tools in numeral methods of ODE systems. The Runge-Kutta-Fehlberg method is an adaptive time stepping that renders a good trade-off between the stability and efficiency. Can we also have an adaptive time stepping for ResNets to ensure both stability and performance? In this study, we analyze the effects of time stepping on the Euler method and ResNets. We establish a stability condition for ResNets with step sizes and weight parameters, and point out the effects of step sizes on the stability and performance. Inspired by our analyses, we develop an adaptive time stepping controller that is dependent on the parameters of the current step, and aware of previous steps. The controller is jointly optimized with the network training so that variable step sizes and evolution time can be adaptively adjusted. We conduct experiments on ImageNet and CIFAR to demonstrate the effectiveness. It is shown that our proposed method is able to improve both stability and accuracy without introducing additional overhead in inference phase.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/06/2020

Automatic Policy Synthesis to Improve the Safety of Nonlinear Dynamical Systems

Learning controllers merely based on a performance metric has been prove...
research
05/02/2022

RANG: A Residual-based Adaptive Node Generation Method for Physics-Informed Neural Networks

Learning solutions of partial differential equations (PDEs) with Physics...
research
03/07/2020

Stability and error estimates for the variable step-size BDF2 method for linear and semilinear parabolic equations

In this paper stability and error estimates for time discretizations of ...
research
03/25/2020

An Energy-based Approach to Ensure the Stability of Learned Dynamical Systems

Non-linear dynamical systems represent a compact, flexible, and robust t...
research
09/14/2021

Learning Based Adaptive Force Control of Robotic Manipulation Based on Real-Time Object Stiffness Detection

Force control is essential for medical robots when touching and contacti...
research
03/28/2020

Long time H^s_α stability of a classical scheme for Cahn-Hilliard equation with polynomial nonlinearity

In this paper we investigate the long time stability of the implicit Eul...
research
08/07/2022

Accelerating Numerical Solvers for Large-Scale Simulation of Dynamical System via NeurVec

Ensemble-based large-scale simulation of dynamical systems is essential ...

Please sign up or login with your details

Forgot password? Click here to reset