High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise

08/17/2022
by   Daniela A. Parletta, et al.
0

In this work we study high probability bounds for stochastic subgradient methods under heavy tailed noise. In this case the noise is only assumed to have finite variance as opposed to a sub-Gaussian distribution for which it is known that standard subgradient methods enjoys high probability bounds. We analyzed a clipped version of the projected stochastic subgradient method, where subgradient estimates are truncated whenever they have large norms. We show that this clipping strategy leads both to near optimal any-time and finite horizon bounds for many classical averaging schemes. Preliminary experiments are shown to support the validity of the method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/21/2020

Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping

In this paper, we propose a new accelerated stochastic first-order metho...
research
03/06/2023

Balancing Cooperativeness and Adaptiveness in the (Noisy) Iterated Prisoner's Dilemma

Ever since Axelrod's seminal work, tournaments served as the main benchm...
research
06/10/2021

Near-Optimal High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

Thanks to their practical efficiency and random nature of the data, stoc...
research
06/17/2020

Nearly Optimal Robust Method for Convex Compositional Problems with Heavy-Tailed Noise

In this paper, we propose robust stochastic algorithms for solving conve...
research
11/24/2022

Theoretical Study of Optimizing Rugged Landscapes with the cGA

Estimation of distribution algorithms (EDAs) provide a distribution - ba...
research
02/14/2022

Stochastic linear optimization never overfits with quadratically-bounded losses on general data

This work shows that a diverse collection of linear optimization methods...
research
03/22/2023

Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises

Recently, several studies consider the stochastic optimization problem b...

Please sign up or login with your details

Forgot password? Click here to reset