Acceleration Methods

01/23/2021
by   Alexandre d'Aspremont, et al.
0

This monograph covers some recent advances on a range of acceleration techniques frequently used in convex optimization. We first use quadratic optimization problems to introduce two key families of methods, momentum and nested optimization schemes, which coincide in the quadratic case to form the Chebyshev method whose complexity is analyzed using Chebyshev polynomials. We discuss momentum methods in detail, starting with the seminal work of Nesterov (1983) and structure convergence proofs using a few master templates, such as that of optimized gradient methods which have the key benefit of showing how momentum methods maximize convergence rates. We further cover proximal acceleration techniques, at the heart of the Catalyst and Accelerated Hybrid Proximal Extragradient frameworks, using similar algorithmic patterns. Common acceleration techniques directly rely on the knowledge of some regularity parameters of the problem at hand, and we conclude by discussing restart schemes, a set of simple techniques to reach nearly optimal convergence rates while adapting to unobserved regularity parameters.

READ FULL TEXT

Authors

page 1

page 2

page 3

page 4

06/01/2020

Factorial Powers for Stochastic Optimization

The convergence rates for convex and non-convex optimization methods dep...
11/11/2021

Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum

Stochastic gradient descent with momentum (SGDM) is the dominant algorit...
02/12/2020

Average-case Acceleration Through Spectral Density Estimation

We develop a framework for designing optimal quadratic optimization meth...
02/23/2017

Convergence acceleration of alternating series

A new simple convergence acceleration method is proposed for a certain w...
01/07/2021

Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization

We extend the Approximate-Proximal Point (aProx) family of model-based m...
05/17/2020

From Proximal Point Method to Nesterov's Acceleration

The proximal point method (PPM) is a fundamental method in optimization ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.