A Contraction Theory Approach to Optimization Algorithms from Acceleration Flows

05/18/2021
by   Pedro Cisneros-Velarde, et al.
0

Much recent interest has focused on the design of optimization algorithms from the discretization of an associated optimization flow, i.e., a system of differential equations (ODEs) whose trajectories solve an associated optimization problem. Such a design approach poses an important problem: how to find a principled methodology to design and discretize appropriate ODEs. This paper aims to provide a solution to this problem through the use of contraction theory. We first introduce general mathematical results that explain how contraction theory guarantees the stability of the implicit and explicit Euler integration methods. Then, we propose a novel system of ODEs, namely the Accelerated-Contracting-Nesterov flow, and use contraction theory to establish it is an optimization flow with exponential convergence rate, from which the linear convergence rate of its associated optimization algorithm is immediately established. Remarkably, a simple explicit Euler discretization of this flow corresponds to the Nesterov acceleration method. Finally, we present how our approach leads to performance guarantees in the design of optimization algorithms for time-varying optimization problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/27/2020

Distributed and time-varying primal-dual dynamics via contraction analysis

In this paper, we provide a holistic analysis of the primal-dual dynamic...
research
02/11/2019

Acceleration via Symplectic Discretization of High-Resolution Differential Equations

We study first-order optimization methods obtained by discretizing ordin...
research
05/15/2023

On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights

We study connections between differential equations and optimization alg...
research
10/01/2021

Contraction Theory for Nonlinear Stability Analysis and Learning-based Control: A Tutorial Overview

Contraction theory is an analytical tool to study differential dynamics ...
research
02/23/2021

Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization

Viewing optimization methods as numerical integrators for ordinary diffe...
research
06/06/2022

Essential convergence rate of ordinary differential equations appearing in optimization

Some continuous optimization methods can be connected to ordinary differ...
research
05/28/2021

Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks

We present a new class of adaptive stochastic optimization algorithms, w...

Please sign up or login with your details

Forgot password? Click here to reset