Log In Sign Up

A Unified Convergence Theorem for Stochastic Optimization Methods

by   Xiao Li, et al.

In this work, we provide a fundamental unified convergence theorem used for deriving expected and almost sure convergence results for a series of stochastic optimization methods. Our unified theorem only requires to verify several representative conditions and is not tailored to any specific algorithm. As a direct application, we recover expected and almost sure convergence results of the stochastic gradient method (SGD) and random reshuffling (RR) under more general settings. Moreover, we establish new expected and almost sure convergence results for the stochastic proximal gradient method (prox-SGD) and stochastic model-based methods (SMM) for nonsmooth nonconvex optimization problems. These applications reveal that our unified theorem provides a plugin-type convergence analysis and strong convergence guarantees for a wide class of stochastic optimization methods.


page 1

page 2

page 3

page 4


A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints

We develop a simple routine unifying the analysis of several important r...

Shampoo: Preconditioned Stochastic Tensor Optimization

Preconditioned gradient methods are among the most general and powerful ...

Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology

We develop a general framework unifying several gradient-based stochasti...

signProx: One-Bit Proximal Algorithm for Nonconvex Stochastic Optimization

Stochastic gradient descent (SGD) is one of the most widely used optimiz...

Data augmentation as stochastic optimization

We present a theoretical framework recasting data augmentation as stocha...

Particle Filtering Methods for Stochastic Optimization with Application to Large-Scale Empirical Risk Minimization

There is a recent interest in developing statistical filtering methods f...

Microscopic Advances with Large-Scale Learning: Stochastic Optimization for Cryo-EM

Determining the 3D structures of biological molecules is a key problem f...