Randomized Smoothing for Stochastic Optimization

03/22/2011
by   John C. Duchi, et al.
0

We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic optimization procedures, both in expectation and with high probability, that have optimal dependence on the variance of the gradient estimates. To the best of our knowledge, these are the first variance-based rates for non-smooth optimization. We give several applications of our results to statistical estimation problems, and provide experimental results that demonstrate the effectiveness of the proposed algorithms. We also describe how a combination of our algorithm with recent work on decentralized optimization yields a distributed stochastic optimization algorithm that is order-optimal.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/05/2018

Accelerated Randomized Coordinate Descent Methods for Stochastic Optimization and Online Learning

We propose accelerated randomized coordinate descent algorithms for stoc...
research
08/04/2015

Asynchronous stochastic convex optimization

We show that asymptotically, completely asynchronous stochastic gradient...
research
06/08/2020

Stochastic Optimization with Non-stationary Noise

We investigate stochastic optimization problems under relaxed assumption...
research
01/07/2021

Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization

We extend the Approximate-Proximal Point (aProx) family of model-based m...
research
12/19/2019

Zeroth-order Stochastic Compositional Algorithms for Risk-Aware Learning

We present Free-MESSAGEp, the first zeroth-order algorithm for convex me...
research
07/18/2012

Stochastic optimization and sparse statistical recovery: An optimal algorithm for high dimensions

We develop and analyze stochastic optimization algorithms for problems i...
research
06/30/2023

An Oblivious Stochastic Composite Optimization Algorithm for Eigenvalue Optimization Problems

In this work, we revisit the problem of solving large-scale semidefinite...

Please sign up or login with your details

Forgot password? Click here to reset