Sharp Analysis of Sketch-and-Project Methods via a Connection to Randomized Singular Value Decomposition

08/20/2022
by   Michał Dereziński, et al.
0

Sketch-and-project is a framework which unifies many known iterative methods for solving linear systems and their variants, as well as further extensions to non-linear optimization problems. It includes popular methods such as randomized Kaczmarz, coordinate descent, variants of the Newton method in convex optimization, and others. In this paper, we obtain sharp guarantees for the convergence rate of sketch-and-project methods via new tight spectral bounds for the expected sketched projection matrix. Our estimates reveal a connection between the sketch-and-project convergence rate and the approximation error of another well-known but seemingly unrelated family of algorithms, which use sketching to accelerate popular matrix factorizations such as QR and SVD. This connection brings us closer to precisely quantifying how the performance of sketch-and-project solvers depends on their sketch size. Our analysis covers not only Gaussian and sub-gaussian sketching matrices, but also a family of efficient sparse sketching methods known as LESS embeddings. Our experiments back up the theory and demonstrate that even extremely sparse sketches show the same convergence properties in practice.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/11/2021

Randomized Extended Kaczmarz is a Limit Point of Sketch-and-Project

The sketch-and-project (SAP) framework for solving systems of linear equ...
research
06/06/2023

A sketch-and-project method for solving the matrix equation AXB = C

In this paper, based on an optimization problem, a sketch-and-project me...
research
03/07/2019

A Rank-1 Sketch for Matrix Multiplicative Weights

We show that a simple randomized sketch of the matrix multiplicative wei...
research
05/22/2023

Sketch-and-Project Meets Newton Method: Global 𝒪(k^-2) Convergence with Low-Rank Updates

In this paper, we propose the first sketch-and-project Newton method wit...
research
07/15/2022

PLSS: A Projected Linear Systems Solver

We propose iterative projection methods for solving square or rectangula...
research
02/21/2020

Optimal Randomized First-Order Methods for Least-Squares Problems

We provide an exact analysis of a class of randomized algorithms for sol...
research
02/01/2019

Compressing Gradient Optimizers via Count-Sketches

Many popular first-order optimization methods (e.g., Momentum, AdaGrad, ...

Please sign up or login with your details

Forgot password? Click here to reset