ALS: Augmented Lagrangian Sketching Methods for Linear Systems

08/12/2022
by   Md Sarowar Morshed, et al.
0

We develop two fundamental stochastic sketching techniques; Penalty Sketching (PS) and Augmented Lagrangian Sketching (ALS) for solving consistent linear systems. The proposed PS and ALS techniques extend and generalize the scope of Sketch Project (SP) method by introducing Lagrangian penalty sketches. In doing so, we recover SP methods as special cases and furthermore develop a family of new stochastic iterative methods. By varying sketch parameters in the proposed PS method, we recover novel stochastic methods such as Penalty Newton Descent, Penalty Kaczmarz, Penalty Stochastic Descent, Penalty Coordinate Descent, Penalty Gaussian Pursuit, and Penalty Block Kaczmarz. Furthermore, the proposed ALS method synthesizes a wide variety of new stochastic methods such as Augmented Newton Descent, Augmented Kaczmarz, Augmented Stochastic Descent, Augmented Coordinate Descent, Augmented Gaussian Pursuit, and Augmented Block Kaczmarz into one framework. Moreover, we show that the developed PS and ALS frameworks can be used to reformulate the original linear system into equivalent stochastic optimization problems namely the Penalty Stochastic Reformulation and Augmented Stochastic Reformulation. We prove global convergence rates for the PS and ALS methods as well as sub-linear 𝒪(1/k) rates for the Cesaro average of iterates. The proposed convergence results hold for a wide family of distributions of random matrices, which provides the opportunity of fine-tuning the randomness of the method suitable for specific applications. Finally, we perform computational experiments that demonstrate the efficiency of our methods compared to the existing SP methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/23/2022

Augmented Newton Method for Optimization: Global Linear Rate and Momentum Interpretation

We propose two variants of Newton method for solving unconstrained minim...
research
09/15/2020

Training neural networks under physical constraints using a stochastic augmented Lagrangian approach

We investigate the physics-constrained training of an encoder-decoder ne...
research
05/17/2022

Penalty Augmented Kaczmarz Methods For Linear Systems Linear Feasibility Problems

In this work, we shed light on the so-called Kaczmarz method for solving...
research
12/24/2020

Stochastic Steepest Descent Methods for Linear Systems: Greedy Sampling Momentum

Recently proposed adaptive Sketch Project (SP) methods connect sever...
research
08/30/2013

Separable Approximations and Decomposition Methods for the Augmented Lagrangian

In this paper we study decomposition methods based on separable approxim...
research
06/04/2017

Stochastic Reformulations of Linear Systems: Algorithms and Convergence Theory

We develop a family of reformulations of an arbitrary consistent linear ...
research
01/13/2022

Consistent Approximations in Composite Optimization

Approximations of optimization problems arise in computational procedure...

Please sign up or login with your details

Forgot password? Click here to reset