Improving the Efficiency of Gradient Descent Algorithms Applied to Optimization Problems with Dynamical Constraints

08/26/2022
by   Ion Matei, et al.
0

We introduce two block coordinate descent algorithms for solving optimization problems with ordinary differential equations (ODEs) as dynamical constraints. The algorithms do not need to implement direct or adjoint sensitivity analysis methods to evaluate loss function gradients. They results from reformulation of the original problem as an equivalent optimization problem with equality constraints. The algorithms naturally follow from steps aimed at recovering the gradient-decent algorithm based on ODE solvers that explicitly account for sensitivity of the ODE solution. In our first proposed algorithm we avoid explicitly solving the ODE by integrating the ODE solver as a sequence of implicit constraints. In our second algorithm, we use an ODE solver to reset the ODE solution, but no direct are adjoint sensitivity analysis methods are used. Both algorithm accepts mini-batch implementations and show significant efficiency benefits from GPU-based parallelization. We demonstrate the performance of the algorithms when applied to learning the parameters of the Cucker-Smale model. The algorithms are compared with gradient descent algorithms based on ODE solvers endowed with sensitivity analysis capabilities, for various number of state size, using Pytorch and Jax implementations. The experimental results demonstrate that the proposed algorithms are at least 4x faster than the Pytorch implementations, and at least 16x faster than Jax implementations. For large versions of the Cucker-Smale model, the Jax implementation is thousands of times faster than the sensitivity analysis-based implementation. In addition, our algorithms generate more accurate results both on training and test data. Such gains in computational efficiency is paramount for algorithms that implement real time parameter estimations, such as diagnosis algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/30/2023

Projection-based first-order constrained optimization solver for robotics

Robot programming tools ranging from inverse kinematics (IK) to model pr...
research
10/23/2019

Sensitivity-based Heuristic for Guaranteed Global Optimization with Nonlinear Ordinary Differential Equations

We focus on interval algorithms for computing guaranteed enclosures of t...
research
09/23/2022

A new perspective on parameter study of optimization problems

We provide a new perspective on the study of parameterized optimization ...
research
08/01/2023

Krylov Solvers for Interior Point Methods with Applications in Radiation Therapy

Interior point methods are widely used for different types of mathematic...
research
12/05/2021

A Novel Sequential Coreset Method for Gradient Descent Algorithms

A wide range of optimization problems arising in machine learning can be...
research
06/02/2022

PNODE: A memory-efficient neural ODE framework based on high-level adjoint differentiation

Neural ordinary differential equations (neural ODEs) have emerged as a n...
research
10/22/2022

torchode: A Parallel ODE Solver for PyTorch

We introduce an ODE solver for the PyTorch ecosystem that can solve mult...

Please sign up or login with your details

Forgot password? Click here to reset