Convergence Analysis of Optimization Algorithms

07/06/2017

∙

The regret bound of an optimization algorithms is one of the basic criteria for evaluating the performance of the given algorithm. By inspecting the differences between the regret bounds of traditional algorithms and adaptive one, we provide a guide for choosing an optimizer with respect to the given data set and the loss function. For analysis, we assume that the loss function is convex and its gradient is Lipschitz continuous.

READ FULL TEXT

Convergence Analysis of Optimization Algorithms

Optimizing Optimizers: Regret-optimal gradient descent algorithms

Learning to Optimize in Swarms

Hemingway: Modeling Distributed Optimization Algorithms

Accelerating Optimization via Adaptive Prediction

New optimization algorithms for neural network training using operator splitting techniques

Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizers

Online Convex Optimization Perspective for Learning from Dynamically Revealed Preferences

Convergence Analysis of Optimization Algorithms

Related Research

Optimizing Optimizers: Regret-optimal gradient descent algorithms

Learning to Optimize in Swarms

Hemingway: Modeling Distributed Optimization Algorithms

Accelerating Optimization via Adaptive Prediction

New optimization algorithms for neural network training using operator splitting techniques

Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizers

Online Convex Optimization Perspective for Learning from Dynamically Revealed Preferences