A Nearly-Optimal Bound for Fast Regression with ℓ_∞ Guarantee

02/01/2023
by   Zhao Song, et al.
0

Given a matrix A∈ℝ^n× d and a vector b∈ℝ^n, we consider the regression problem with ℓ_∞ guarantees: finding a vector x'∈ℝ^d such that x'-x^*_∞≤ϵ/√(d)·Ax^*-b_2·A^† where x^*=min_x∈ℝ^dAx-b_2. One popular approach for solving such ℓ_2 regression problem is via sketching: picking a structured random matrix S∈ℝ^m× n with m≪ n and SA can be quickly computed, solve the “sketched” regression problem min_x∈ℝ^dSAx-Sb_2. In this paper, we show that in order to obtain such ℓ_∞ guarantee for ℓ_2 regression, one has to use sketching matrices that are dense. To the best of our knowledge, this is the first user case in which dense sketching matrices are necessary. On the algorithmic side, we prove that there exists a distribution of dense sketching matrices with m=ϵ^-2dlog^3(n/δ) such that solving the sketched regression problem gives the ℓ_∞ guarantee, with probability at least 1-δ. Moreover, the matrix SA can be computed in time O(ndlog n). Our row count is nearly-optimal up to logarithmic factors, and significantly improves the result in [Price, Song and Woodruff, ICALP'17], in which a super-linear in d rows, m=Ω(ϵ^-2d^1+γ) for γ=Θ(√(loglog n/log d)) is required. We also develop a novel analytical framework for ℓ_∞ guarantee regression that utilizes the Oblivious Coordinate-wise Embedding (OCE) property introduced in [Song and Yu, ICML'21]. Our analysis is arguably much simpler and more general than [Price, Song and Woodruff, ICALP'17], and it extends to dense sketches for tensor product of vectors.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/08/2022

Dynamic Tensor Product Regression

In this work, we initiate the study of Dynamic Tensor Product Regression...
research
03/14/2022

Fast Regression for Structured Inputs

We study the ℓ_p regression problem, which requires finding 𝐱∈ℝ^d that m...
research
01/14/2021

Minimum Cost Flows, MDPs, and ℓ_1-Regression in Nearly Linear Time for Dense Instances

In this paper we provide new randomized algorithms with improved runtime...
research
09/03/2019

Almost Optimal Tensor Sketch

We construct a matrix M∈ R^m⊗ d^c with just m=O(c λ ε^-2polylog1/εδ) row...
research
11/11/2013

Learning Mixtures of Linear Classifiers

We consider a discriminative learning (regression) problem, whereby the ...
research
02/25/2017

Efficient coordinate-wise leading eigenvector computation

We develop and analyze efficient "coordinate-wise" methods for finding t...
research
06/24/2021

Johnson-Lindenstrauss Embeddings with Kronecker Structure

We prove the Johnson-Lindenstrauss property for matrices Φ D_ξ where Φ h...

Please sign up or login with your details

Forgot password? Click here to reset