Convergence Analysis of Inexact Randomized Iterative Methods

03/19/2019
by   Nicolas Loizou, et al.
11

In this paper we present a convergence rate analysis of inexact variants of several randomized iterative methods. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic subspace ascent. A common feature of these methods is that in their update rule a certain sub-problem needs to be solved exactly. We relax this requirement by allowing for the sub-problem to be solved inexactly. In particular, we propose and analyze inexact randomized iterative methods for solving three closely related problems: a convex stochastic quadratic optimization problem, a best approximation problem and its dual, a concave quadratic maximization problem. We provide iteration complexity results under several assumptions on the inexactness error. Inexact variants of many popular and some more exotic methods, including randomized block Kaczmarz, randomized Gaussian Kaczmarz and randomized block coordinate descent, can be cast as special cases. Numerical experiments demonstrate the benefits of allowing inexactness.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/19/2013

Inexact Coordinate Descent: Complexity and Preconditioning

In this paper we consider the problem of minimizing a convex function us...
research
02/21/2020

Stochastic Subspace Cubic Newton Method

In this paper, we propose a new randomized second-order optimization alg...
research
09/25/2020

Randomized Progressive Hedging methods for Multi-stage Stochastic Programming

Progressive Hedging is a popular decomposition algorithm for solving mul...
research
05/19/2016

Randomized Primal-Dual Proximal Block Coordinate Updates

In this paper we propose a randomized primal-dual proximal block coordin...
research
10/04/2015

Implicit stochastic approximation

The need to carry out parameter estimation from massive data has reinvig...
research
05/20/2019

Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols

In this work we present a new framework for the analysis and design of r...
research
10/11/2021

Randomized Extended Kaczmarz is a Limit Point of Sketch-and-Project

The sketch-and-project (SAP) framework for solving systems of linear equ...

Please sign up or login with your details

Forgot password? Click here to reset