Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions

02/03/2019
by   Adrien Taylor, et al.
0

We provide a novel computer-assisted technique for systematically analyzing first-order methods for optimization. In contrast with previous works, the approach is particularly suited for handling sublinear convergence rates and stochastic oracles. The technique relies on semidefinite programming and potential functions. It allows simultaneously obtaining worst-case guarantees on the behavior of those algorithms, and assisting in choosing appropriate parameters for tuning their worst-case performances. The technique also benefits from comfortable tightness guarantees, meaning that unsatisfactory results can be improved only by changing the setting. We use the approach for analyzing deterministic and stochastic first-order methods under different assumptions on the nature of the stochastic noise. Among others, we treat unstructured noise with bounded variance, different noise models arising in over-parametrized expectation minimization problems, and randomized block-coordinate descent schemes.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/11/2022

PEPit: computer-assisted worst-case analyses of first-order optimization methods in Python

PEPit is a Python package aiming at simplifying the access to worst-case...
research
06/03/2019

A Generic Acceleration Framework for Stochastic Composite Optimization

In this paper, we introduce various mechanisms to obtain accelerated fir...
research
05/25/2020

Boosting First-order Methods by Shifting Objective: New Schemes with Faster Worst Case Rates

We propose a new methodology to design first-order methods for unconstra...
research
06/08/2020

Stochastic Optimization with Non-stationary Noise

We investigate stochastic optimization problems under relaxed assumption...
research
05/25/2022

A systematic approach to Lyapunov analyses of continuous-time models in convex optimization

First-order methods are often analyzed via their continuous-time models,...
research
03/11/2022

Automatic Performance Estimation for Decentralized Optimization

We present a methodology to automatically compute worst-case performance...
research
04/02/2021

Defending Against Image Corruptions Through Adversarial Augmentations

Modern neural networks excel at image classification, yet they remain vu...

Please sign up or login with your details

Forgot password? Click here to reset