Optimizing Optimizers: Regret-optimal gradient descent algorithms

12/31/2020
by   Philippe Casgrain, et al.
0

The need for fast and robust optimization algorithms are of critical importance in all areas of machine learning. This paper treats the task of designing optimization algorithms as an optimal control problem. Using regret as a metric for an algorithm's performance, we derive the necessary and sufficient dynamics that regret-optimal algorithms must satisfy as a discrete-time difference equation. We study the existence, uniqueness and consistency of regret-optimal algorithms and derive bounds on rates of convergence to solutions of convex optimization problems. Though closed-form optimal dynamics cannot be obtained in general, we present fast numerical methods for approximating them, generating optimization algorithms which directly optimize their long-term regret. Lastly, these are benchmarked against commonly used optimization algorithms to demonstrate their effectiveness.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/06/2017

Convergence Analysis of Optimization Algorithms

The regret bound of an optimization algorithms is one of the basic crite...
research
02/25/2010

Less Regret via Online Conditioning

We analyze and evaluate an online gradient descent algorithm with adapti...
research
11/22/2021

No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimization

We develop an algorithmic framework for solving convex optimization prob...
research
12/23/2015

Adaptive Algorithms for Online Convex Optimization with Long-term Constraints

We present an adaptive online gradient descent algorithm to solve online...
research
12/08/2021

COSMIC: fast closed-form identification from large-scale data for LTV systems

We introduce a closed-form method for identification of discrete-time li...
research
03/21/2020

A new regret analysis for Adam-type algorithms

In this paper, we focus on a theory-practice gap for Adam and its varian...
research
12/16/2019

A Control-Theoretic Perspective on Optimal High-Order Optimization

In this paper, we provide a control-theoretic perspective on optimal ten...

Please sign up or login with your details

Forgot password? Click here to reset