Optimal Randomized First-Order Methods for Least-Squares Problems

02/21/2020
by   Jonathan Lacotte, et al.
0

We provide an exact analysis of a class of randomized algorithms for solving overdetermined least-squares problems. We consider first-order methods, where the gradients are pre-conditioned by an approximation of the Hessian, based on a subspace embedding of the data matrix. This class of algorithms encompasses several randomized methods among the fastest solvers for least-squares problems. We focus on two classical embeddings, namely, Gaussian projections and subsampled randomized Hadamard transforms (SRHT). Our key technical innovation is the derivation of the limiting spectral density of SRHT embeddings. Leveraging this novel result, we derive the family of normalized orthogonal polynomials of the SRHT density and we find the optimal pre-conditioned first-order method along with its rate of convergence. Our analysis of Gaussian embeddings proceeds similarly, and leverages classical random matrix theory results. In particular, we show that for a given sketch size, SRHT embeddings exhibits a faster rate of convergence than Gaussian embeddings. Then, we propose a new algorithm by optimizing the computational complexity over the choice of the sketching dimension. To our knowledge, our resulting algorithm yields the best known complexity for solving least-squares problems with no condition number dependence.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/06/2019

Faster Least Squares Optimization

We investigate randomized methods for solving overdetermined linear leas...
research
06/10/2020

Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization

We propose a new randomized algorithm for solving L2-regularized least-s...
research
02/03/2020

Limiting Spectrum of Randomized Hadamard Transform and Optimal Iterative Sketching Methods

We provide an exact analysis of the limiting spectrum of matrices random...
research
08/20/2022

Sharp Analysis of Sketch-and-Project Methods via a Connection to Randomized Singular Value Decomposition

Sketch-and-project is a framework which unifies many known iterative met...
research
07/15/2021

Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update

In second-order optimization, a potential bottleneck can be computing th...
research
10/16/2015

Robust Partially-Compressed Least-Squares

Randomized matrix compression techniques, such as the Johnson-Lindenstra...
research
07/20/2023

A Framework for Statistical Inference via Randomized Algorithms

Randomized algorithms, such as randomized sketching or projections, are ...

Please sign up or login with your details

Forgot password? Click here to reset