Sharpened Error Bounds for Random Sampling Based ℓ_2 Regression

1 Introduction

Given $n$ data instances $x^{(1)}, \dots, x^{(n)}$ , each of dimension $d$ , and $n$ responses $y_{1}, \dots, y_{n}$ , it is interesting to find a model $\boldmathβ\unboldmath∈Rd$ such that $y=X\boldmathβ\unboldmath$ . If $n > d$ , there will not in general exist a solution to the linear system, so we instead seek to find a model $\boldmathβ\unboldmathlsr$ such that $y≈X\boldmathβ\unboldmathlsr$ . This can be formulated as the least squares regression (LSR) problem:

\boldmathβ\unboldmathlsr=argmin\boldmathβ\unboldmath∈Rd∥∥y−X\boldmathβ\unboldmath∥∥22.

(1)

Suppose $n \geq d$ , in general it takes $O (n d^{2})$ time and $O (n d)$ space to compute $\boldmathβ\unboldmathlsr$

using the Cholesky decomposition, the QR decomposition, or the singular value decomposition (SVD)

[9].

LSR is perhaps one of the most widely used method in data processing, however, solving LSR for big data is very time and space expensive. In the big-data problems where $n$ and $d$ are both large, the $O (n d^{2})$ time complexity and $O (n d)$ space complexity make LSR prohibitive. So it is of great interest to find efficient solution to the LSR problem. Fortunately, when $n ≫ d$ , one can use a small portion of the $n$ instances instead of using the full data to approximately compute $\boldmathβ\unboldmathlsr$ , and the computation cost can thereby be significantly reduced. Random sampling based methods [5, 11, 12] and random projection based methods [2, 6] have been applied to make LSR more efficiently solved.

Formally speaking, let $S \in R^{c \times n}$ be a random sampling/projection matrix, we solve the following problem instead of (1):

~\boldmathβ\unboldmathS=argmin\boldmathβ\unboldmath∈Rd∥∥Sy−SX\boldmathβ\unboldmath∥∥22.

(2)

This problem can be solved in only $O (c d^{2})$ time and $O (c d)$ space. If the random sampling/projection matrix $S$ is constructed using some special techniques, then it is ensured theoretically that $~\boldmathβ\unboldmathS≈\boldmathβ\unboldmathlsr$ and that

∥∥y−X~\boldmathβ\unboldmathS∥∥22≤(1+ϵ)∥∥y−X% \boldmathβ\unboldmathlsr∥∥22

(3)

hold with high probability. There are two criteria to evaluate random sampling/projection based LSR.

Running Time. That is, the total time complexity in constructing $S \in R^{c \times n}$ and computing $S X \in R^{c \times d}$ .
Dimension after Projection. Given an error parameter $ϵ$ , we hope that there exists a polynomial function $C (d, ϵ)$ such that if $c > C (d, ϵ)$ , the inequality (3) holds with high probability for all $X \in R^{n \times d}$ and $y \in R^{n}$ . Obviously $C (d, ϵ)$ is the smaller the better because the induced problem (2) can be solved in less time and space if $c$ is small.

1.1 Contributions

The leverage scores based sampling is an important random sampling technique widely studied and empirically evaluated in the literature [2, 4, 5, 8, 11, 13, 12, 15, 16]. When applied to accelerate LSR, error analysis of the leverage scores based sampling is available in the literature. It was shown in [4] showed that by using the leverage scores based sampling with replacement, when

c \geq O (d^{2} ϵ^{- 2}),

the inequality (3) holds with high probability. Later on, [5] showed that by using the leverage scores based sampling without replacement,

c \geq O (d ϵ^{- 2} log d)

is sufficient to make (3) hold with high probability. In Theorem 3.1 we show that (3) holds with high probability when

c \geq O (d log d + d ϵ^{- 1}),

using the same leverage scores based sampling without replacement. Our results are described in Theorem 3.1. Our proof techniques are based on the previous work [5, 6], and our proof is self-contained.

Though uniform sampling can have very bad worst-case performance when applied to the LSR problem [11], it is still the simplest and most efficient strategy for column sampling. Though the uniform sampling is uniformly worse than the leverage score based sampling from an algorithmic perspective, it is not true from a statistical perspective [11]. Furthermore, when the leverage scores of $X$ are very homogeneous, the uniform sampling has virtually the same performance as the leverage scores based sampling [11]. So uniform sampling is still worth of study. We provide in Theorem 3.2 an error bound for the uniform sampling based LSR. We show that when

c > O (μ d ϵ^{- 2} log d),

the uniform sampling based LSR attains a $2 + ϵ$ bound with positive probability. Here $μ$ denotes the matrix coherence of $X$ .

1: Input: an

n \times d

real matrix

X

, target dimension

c < n

2: (Exactly or approximately) compute the leverage scores of

X

l_{1}, \dots, l_{n}

;

3: Compute the sampling probabilities by

p_{i} = min {1, c l_{i} / d}

for

i = 1

n

;

4: Denote the set containing the indices of selected rows by

C

, initialized by

\emptyset

;

5: For each index

i \in [n]

, add

i

C

with probability

p_{i}

;

6: Compute the diagonal matrix

D = d i a g (p_{1}^{- 1}, \dots, p_{n}^{- 1})

;

7: return

S ⟵

the rows of

D

indexed by

C

Algorithm 1 The Leverage Scores Based Sampling (without Replacement).

2 Preliminaries and Previous Work

For a matrix $X = [x_{i j}] \in R^{n \times d}$ , we let $x^{(i)}$ be its $i$ -th row, $x_{j}$ be its $j$ -th column, $∥ X ∥_{F} = (\sum_{i, j} x_{i j}^{2})^{1 / 2}$ be its Frobenius norm, and $∥ X ∥_{2} = {max}_{∥ z ∥_{2} = 1} ∥ X z ∥_{2}$ be its spectral norm. We let $I_{n}$ be an $n \times n$ identity matrix and let $0$

be an all-zero matrix with proper size.

We let the thin singular value decomposition of $X \in R^{n \times d}$ be

X=UX\boldmathΣ\unboldmathXVTX=d∑i=1σi(X)uX,ivTX,i.

Here $U_{X}$ , $\boldmathΣ\unboldmathX$ , and $V_{X}$ are of sizes $n \times d$ , $d \times d$ , and $d \times d$ , and the singular values $σ_{1} (X), \dots, σ_{d} (X)$ are in non-increasing order. We let $U_{X}^{⊥}$ be an $n \times (n - d)$

column orthogonal matrix such that

U_{X}^{T} U_{X}^{⊥} = 0

. The condition number of

X

is defined by

κ (X) = σ_{max} (X) / σ_{min} (X)

Based on SVD, the (row) statistical leverage scores of $X \in R^{n \times d}$ is defined by

l_{i} = ∥ ∥ u_{X}^{(i)} {∥ ∥}_{2}^{2}, i = 1, \dots, n,

where $u_{X}^{(i)}$ is the $i$ -th row of $U_{X}$ . It is obvious that $\sum_{i = 1}^{n} l_{i} = d$ . Exactly computing the $n$ leverages scores costs $O (n d^{2})$ time, which is as expensive as exactly solving the LSR problem (1). Fortunately, if $X$ is a skinny matrix, the leverages scores can be highly efficiently computed within arbitrary accuracy using the techniques of [2, 3].

There are many ways to construct the random sampling/projection matrix $S$ , and below we describe some of them.

Uniform Sampling. The sampling matrix $S$ is constructed by sampling $c$ rows of the identity matrix $I_{n}$ uniformly at random. This method is the simplest and fastest, but in the worst case its performance is very bad [11].
Leverage Scores Based Sampling. The sampling matrix $S$ is computed by Algorithm 1; $S$ has $c$ rows in expectation. This method is proposed in [4, 5].
Subsampled Randomized Hadamard Transform (SRHT). The random projection matrix $S = \sqrt{n / c} R H D$ is called SRHT [1, 6, 14] if
- $R \in R^{c \times n}$ is a subset of $c$ rows from the $n \times n$ identity matrix, where the rows are chosen uniformly at random and without replacement;
- $H \in R^{n \times n}$ is a normalized Walsh–Hadamard matrix;
- $D$ is an $n \times n$ random diagonal matrix with each diagonal entry independently chosen to be $+ 1$ or $- 1$ with equal probability.
SRHT is a fast version of the Johnson-Lindenstrauss transform. The performance of SRHT based LSR is analyzed in [6].
Sparse Embedding Matrices. The sparse embedding matrix $S=\boldmathΦ\unboldmathD$ enables random projection performed in time only linear in the number of nonzero entries of $X$ [2]. The random linear map $S=\boldmathΦ\unboldmathD$ is defined by
- $h : [n] \mapsto [c]$ is a random map so that for each $i \in [n]$ , $h (i) = t$ for $t \in [c]$ with probability $1 / c$ ;
- $\boldmathΦ\unboldmath∈{0,1}c×n$ is a $c \times n$ binary matrix with $Φ_{h (i), i} = 1$ , and all remaining entries $0$ ;
- $D$ is the same to the matrix $D$ of SRHT.
Sparse embedding matrices based LSR is guaranteed theoretically in [2].

3 Main Results

We provide in Theorem 3.1 an improved error bound for the leverage scores sampling based LSR.

Theorem 3.1 (The Leverage Score Based Sampling)

Use the leverage score based sampling without replacement (Algorithm 1) to construct the $c \times d$ sampling matrix $S$ where

c \geq O (d ln d + d ϵ^{- 1}),

and solve the approximate LSR problem (2) to obtain $~\boldmathβ\unboldmathS$ . Then with probability at least $0.8$ the following inequalities hold:

	$∥∥y−X~\boldmathβ\unboldmathS∥∥22$	$\leq$	$(1+ϵ)∥∥y−X\boldmathβ% \unboldmathlsr∥∥22,$
	$∥∥\boldmathβ\unboldmathlsr−~\boldmathβ\unboldmathS∥∥22$	$\leq$	$ϵσ2min(X)∥∥y−X\boldmathβ\unboldmathlsr∥∥22≤ϵκ2(X)(γ−2−1)∥∥% \boldmathβ\unboldmathlsr∥∥22,$

where $γ$ is defined by $γ \leq ∥ U_{X} U_{X}^{T} y ∥_{2} / ∥ y ∥_{2} \leq 1$ .

We show in Theorem 3.2 an error bound for the uniform sampling based LSR.

Theorem 3.2 (Uniform Sampling)

Use the uniform sampling without replacement to sample

c \geq 1000 μ d (ln d + 7)

rows of $X$ and compute the approximate LSR problem (2) to obtain $~\boldmathβ\unboldmathS$ . Then with probability $0.05$ the following inequalities hold:

	$∥y−X~\boldmathβ\unboldmathS∥22$	$\leq$	$2.2∥y−X\boldmathβ\unboldmathlsr∥22,$
	$∥∥\boldmathβ\unboldmathlsr−~\boldmathβ\unboldmathS∥∥22$	$\leq$	$1.2σ2min(X)∥∥y−X\boldmathβ\unboldmathlsr∥∥22≤1.2κ2(X)(γ−2−1)∥∥\boldmathβ\unboldmathlsr∥∥22,$

where $γ$ is defined by $γ \leq ∥ U_{X} U_{X}^{T} y ∥_{2} / ∥ y ∥_{2} \leq 1$ .

Since computing $∥y−X~\boldmathβ\unboldmathS∥2$ costs only $O (n d)$ time, so one can repeat the procedure $t$ times and choose the solution that attains the minimal error $∥y−X~\boldmathβ\unboldmathS∥2$ . In this way, the error bounds hold with probability $1 - {0.95}^{t}$ which can be arbitrarily high.

4 Proof

In Section 4.1 we list some of the previous work that will be used in our proof. In Section 4.2 we prove Theorem 3.1. We prove the theorem by using the techniques in the proof of Lemma 1 and 2 of [5] and Lemma 1 and 2 of [6]. For the sake of self-contain, we repeat some of the proof of [5] in our proof. In Section 4.3 we prove Theorem 3.2 using the techniques in [6, 7, 14].

4.1 Key Lemmas

Lemma 1 is a deterministic error bound for the sampling/projection based LSR, which will be used to prove both of Theorem 3.1 and Theorem 3.2

. The random matrix multiplication bounds in Lemma

2 will be used to prove Theorem 3.1. The matrix variable tail bounds in Lemma 3 will be used to prove Theorem 3.2.

Lemma 1 (Deterministic Error Bound, Lemma 1 and 2 of [6])

Suppose we are given an overconstrained least squares approximation problem with $X \in R^{n \times d}$ and $y \in R^{n}$ . We let $\boldmathβ\unboldmathlsr$ be defined in (1) and $~\boldmathβ\unboldmathS$ be defined in (2), and define $z_{S} \in R^{d}$ such that $UXzS=X(\boldmathβ\unboldmathlsr−~\boldmathβ\unboldmathS)$ . Then the following equality and inequalities hold deterministically:

$∥∥y−X~\boldmathβ\unboldmathS∥∥22$	$=$	$∥∥y−X\boldmathβ\unboldmathlsr∥∥22+∥∥UXzS∥∥22,$
$∥∥\boldmathβ\unboldmathlsr−~\boldmathβ\unboldmathS∥∥22$	$\leq$	$\frac{∥ U_{X} z_{S} ∥_{2}^{2}}{σ_{min}^{2} (X)},$
$∥ ∥ z_{S} {∥ ∥}_{2}$	$\leq$	$\frac{∥ ∥ U_{X}^{T} S^{T} S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}}{σ_{min}^{2} (S U_{X})} .$

By further assuming that $∥ U_{X} U_{X}^{T} y ∥_{2} \geq γ ∥ y ∥_{2}$ , it follows that

Proof

The equality and the first two inequalities follow from Lemma 1 of [6]. The last inequality follows from Lemma 2 of [6].

Lemma 2 (Theorem 7 of [5])

Suppose $X \in R^{d \times n}$ , $Y \in R^{n \times p}$ , and $c \leq n$ , and we let $S \in R^{c \times n}$ be the sampling matrix computed by Algorithm 1 taking $X$ and $c$ as input, then

	$E ∥ ∥ X^{T} Y - X^{T} S^{T} S Y {∥ ∥}_{F}$	$\leq$	$\frac{1}{\sqrt{c}} ∥ ∥ X {∥ ∥}_{F} ∥ ∥ Y {∥ ∥}_{F},$
	$E ∥ ∥ X^{T} X - X^{T} S^{T} S X {∥ ∥}_{F}$	$\leq$	$O (\sqrt{\frac{log c}{c}}) ∥ ∥ X {∥ ∥}_{2} ∥ ∥ X {∥ ∥}_{F} .$

Lemma 3 (Theorem 2.2 of [14])

Let $W$ be a finite set of positive semidefinite matrices with dimension $d$ , and suppose that

max W \in W λ_{max} (W) \leq R .

Sample $W_{1}, \dots, W_{c}$ uniformly at random from $W$ without replacement. We define $ξ_{min} = c λ_{min} (E W_{1})$ and $ξ_{max} = c λ_{max} (E W_{1})$ . Then for any $θ_{1} \in (0, 1]$ and $θ_{2} > 1$ , the following inequalities hold:

	$P {λ_{min} (c \sum i = 1 W_{i}) \leq θ_{1} ξ_{min}}$	$\leq$
	$P {λ_{max} (c \sum i = 1 W_{i}) \geq θ_{2} ξ_{max}}$	$\leq$

4.2 Proof of Theorem 3.1

Proof

We first bound the term $σ_{min}^{2}$ as follows. Applying a singular value inequality in [10], we have that for all $i \leq r a n k (X)$

$∣ ∣ 1 - σ_{i}^{2} (S U_{X}) ∣ ∣$	$=$	$∣ ∣ σ_{i} (U_{X}^{T} U_{X}) - σ_{i} (U_{X}^{T} S^{T} S U_{X}) ∣ ∣$
	$\leq$	$σ_{max} (U_{X}^{T} U_{X} - U_{X}^{T} S^{T} S^{T} U_{X})$
	$=$	$∥ ∥ U_{X}^{T} U_{X} - U_{X}^{T} S^{T} S^{T} U_{X} {∥ ∥}_{2} .$

Since the leverage scores of $X$ are also the leverage scores of $U_{X}$ , it follows from Lemma 2 that

E ∥ ∥ U_{X}^{T} U_{X} - U_{X}^{T} S^{T} S^{T} U_{X} {∥ ∥}_{2} \leq O (\sqrt{\frac{ln c}{c}}) ∥ U_{X} ∥_{F} ∥ U_{X} ∥_{2} = O (\sqrt{\frac{d ln c}{c}}) .

It then follows from Markov’s inequality that the inequality

∣ ∣ 1 - σ_{i}^{2} (S U_{X}) ∣ ∣ \leq δ_{1}^{- 1} O (\sqrt{\frac{d ln c}{c}})

holds with probability at least $1 - δ_{1}$ . When

c \geq O (d δ_{1}^{- 2} ϵ_{1}^{- 2} ln (d δ_{1}^{- 2} ϵ_{1}^{- 2})),

(4)

the inequality

σ_{min}^{2} (S U_{X}) \geq 1 - ϵ_{1}

(5)

holds with probability at least $1 - δ_{1}$ .

Now we bound the term $∥ U_{X}^{T} S^{T} S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y ∥_{2}$ . Since $U_{X}^{T} {U_{X}^{⊥}}^{T} = 0$ , we have that

∥ ∥ U_{X}^{T} S^{T} S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}

=

∥ ∥ (U_{X}^{T}) (U_{X}^{⊥} {U_{X}^{⊥}}^{T} y) - (U_{X}^{T}) S^{T} S (U_{X}^{⊥} {U_{X}^{⊥}}^{T} y) {∥ ∥}_{2} .

Since the leverage scores of $X$ are also the leverage scores of $U_{X}$ , it follows from Lemma 2 that

	$E ∥ ∥ (U_{X}^{T}) (U_{X}^{⊥} {U_{X}^{⊥}}^{T} y) - (U_{X}^{T}) S^{T} S (U_{X}^{⊥} {U_{X}^{⊥}}^{T} y) {∥ ∥}_{2}$
	$\leq \frac{1}{\sqrt{c}} ∥ ∥ U_{X} {∥ ∥}_{F} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} = \sqrt{\frac{d}{c}} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} .$

It follows from the Markov’s inequality that the following inequality holds with probability at least $1 - δ_{2}$ :

∥ ∥ U_{X}^{T} S^{T} S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} \leq \frac{δ_{2}^{- 1} \sqrt{d}}{\sqrt{c}} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} .

(6)

Thus when

c \geq d δ_{2}^{- 2} ϵ_{2}^{- 2} (1 - ϵ_{1})^{- 2},

(7)

it follows from (5), (6), and the union bound that the inequality

\frac{∥ ∥ U_{X}^{T} S^{T} S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}}{σ_{min}^{2} (S U_{X})} \leq ϵ_{2} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}

(8)

holds with probability at least $1 - δ_{1} - δ_{2}$ . We let $ϵ_{1} = 0.5$ , $ϵ_{2} = \sqrt{ϵ}$ , $δ_{1} = δ_{2} = 0.1$ , and let $z_{S}$ be defined in Lemma 1. When

c \geq max {O (d ln d), 400 d ϵ^{- 1}},

it follows from (4), (7), (8), and Lemma 1 that with probability at least $0.8$ the following inequality holds:

∥ z_{S} ∥_{2} \leq \sqrt{ϵ} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} .

Since $U⊥XU⊥XTy=y−X\boldmathβ\unboldmathlsr$ , the theorem follows directly from Lemma 1.

4.3 Proof of Theorem 3.2

Proof

We first follow some of the techniques of [7] to bound the two terms

	$σ_{max}^{2} (S U_{X})$	$=$	$σ_{max} (U_{X}^{T} S^{T} S U_{X}),$
	$σ_{min}^{2} (S U_{X})$	$=$	$σ_{min} (U_{X}^{T} S^{T} S U_{X}) .$

We let $u_{i} \in R^{d}$ be the $i$ -th column of $U_{X}^{T}$ , and let $W_{1}, \dots, W_{c}$ be $d \times d$ matrices sampled i.i.d. from ${u_{i} u_{i}^{T}}_{i = 1}^{n}$ uniformly at random without replacement. Obviously, $σ_{k} (U_{X}^{T} S^{T} S U_{X}) = σ_{k} (\sum_{j = 1}^{c} W_{j})$ . We accordingly define

R = max j λ_{max} (W_{j}) = max i ∥ ∥ u_{i} {∥ ∥}_{2}^{2} = \frac{d}{n} μ,

where $μ$ is the row matrix coherence of $X$ , and define

	$ξ_{min}$	$=$	$c λ_{min} (E W_{1}) = \frac{c}{n} λ_{min} (U_{X}^{T} U_{X}) = \frac{c}{n},$
	$ξ_{max}$	$=$	$c λ_{max} (E W_{1}) = \frac{c}{n} λ_{max} (U_{X}^{T} U_{X}) = \frac{c}{n} .$

Then we apply Lemma 3 and obtained the following inequality:

	$P [λ_{min} (c \sum i = 1 W_{i}) \leq \frac{θ_{1} c}{n}]$	$\leq$
	$P [λ_{max} (c \sum i = 1 W_{i}) \geq \frac{θ_{2} c}{n}]$	$\leq$

where $θ_{1} \in (0, 1)$ , $θ_{2} > 1$ , and $δ_{1}, δ_{2} \in (0, 1)$ are arbitrary real numbers. We set

c = max {\frac{μ d ln (d / δ_{1})}{θ_{1} ln θ_{1} - θ_{1} + 1}, \frac{μ d ln (d / δ_{2})}{θ_{2} ln θ_{2} - θ_{2} + 1}},

(9)

it then follows that with probability at least $1 - δ_{1} - δ_{2}$ , both of the following two inequalities hold:

σ_{max} (S U_{X}) \leq \sqrt{\frac{θ_{2} c}{n}} and σ_{min}^{- 2} (S U_{X}) \leq \frac{n}{θ_{1} c} .

(10)

Now we seek to bound the term . Let $C$ be an index set of cardinality $c$ with each element chosen from $[n]$ uniformly at random without replacement, and let $y^{⊥} = U_{X}^{⊥} {U_{X}^{⊥}}^{T} y$ , then we have that

E ∥ ∥ S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}^{2} = E ∥ ∥ S y^{⊥} {∥ ∥}_{2}^{2} = E \sum j \in C (y_{j}^{⊥})^{2} = \frac{c}{n} ∥ ∥ y^{⊥} {∥ ∥}_{2}^{2} .

Thus with probability at least $1 - δ_{3}$ the following inequality holds:

∥ ∥ S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}^{2} \leq \frac{c}{n δ_{3}} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}^{2} .

(11)

Finally, it follows from inequalities (10, 11) and Lemma 1 that

	$∥ z_{S} ∥_{2}$	$\leq$	$\frac{∥ ∥ U_{X}^{T} S^{T} S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}}{σ_{min}^{2} (S U_{X})} \leq \frac{∥ ∥ U_{X}^{T} S^{T} {∥ ∥}_{2} ∥ ∥ S U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}}{σ_{min}^{2} (S U_{X})}$
		$\leq$	$\sqrt{\frac{θ_{2} c}{n}} \frac{n}{θ_{1} c} \sqrt{\frac{c}{} n δ_{3}} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} = \frac{1}{θ_{1}} \sqrt{\frac{θ_{2}}{δ_{3}}} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2} .$

Here the first two inequalities hold deterministically, and the third inequality holds with probability at least $1 - δ_{1} - δ_{2} - δ_{3}$ .

We set $θ_{1} = 1 - ϵ$ , $θ_{2} = 1 + ϵ$ , $δ_{1} = δ_{2} = δ$ , and $δ_{3} = 1 - 3 δ$ . Since $ln θ \approx θ - 1$ when $θ$ is close to $1$ , it follows from (9) that when $c > μ d ϵ^{- 2} (ln d - ln δ)$ , the inequality

∥ ∥ z_{S} {∥ ∥}_{2}^{2} \leq \frac{1 + ϵ}{(1 - ϵ)^{2} (1 - 3 δ)} ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}^{2}

holds with probability at least $δ$ .

Setting $θ_{1} = 0.9556$ , $θ_{2} = 1.045$ , $δ_{1} = δ_{2} = 0.0015$ , and $δ_{3} = 0.947$ , we conclude that when $c = 1000 μ d (ln d + 7)$ , the inequality $∥ z_{S} ∥_{2}^{2} \leq 1.2 ∥ ∥ U_{X}^{⊥} {U_{X}^{⊥}}^{T} y {∥ ∥}_{2}^{2}$ holds with probability at least $0.05$ . Then the theorem follows directly from Lemma 1.

References

[1] C. Boutsidis and A. Gittens. Improved matrix algorithms via the subsampled randomized hadamard transform. SIAM Journal on Matrix Analysis and Applications, 34(3):1301–1340, 2013.
[2] K. L. Clarkson and D. P. Woodruff. Low rank approximation and regression in input sparsity time. In
Annual ACM Symposium on theory of computing (STOC)
. ACM, 2013.
[3] P. Drineas, M. Magdon-Ismail, M. W. Mahoney, and D. P. Woodruff. Fast approximation of matrix coherence and statistical leverage.
Journal of Machine Learning Research
, 13:3441–3472, 2012.
[4] P. Drineas, M. W. Mahoney, and S. Muthukrishnan. Sampling algorithms for $ℓ_{2}$ regression and applications. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm. ACM, 2006.
[5] P. Drineas, M. W. Mahoney, and S. Muthukrishnan. Relative-error CUR matrix decompositions. SIAM Journal on Matrix Analysis and Applications, 30(2):844–881, Sept. 2008.
[6] P. Drineas, M. W. Mahoney, S. Muthukrishnan, and T. Sarlós. Faster least squares approximation. Numerische Mathematik, 117(2):219–249, 2011.
[7] A. Gittens. The spectral norm error of the naive Nyström extension. arXiv preprint arXiv:1110.5305, 2011.
[8] A. Gittens and M. W. Mahoney. Revisiting the nyström method for improved large-scale machine learning. In International Conference on Machine Learning (ICML), 2013.
[9] G. H. Golub and C. F. van Loan. Matrix computations. The Johns Hopkins, 1996.
[10] R. A. Horn and C. R. Johnson. Topics in matrix analysis. 1991. Cambridge University Presss, Cambridge.
[11] P. Ma, M. W. Mahoney, and B. Yu. A statistical perspective on algorithmic leveraging. In International Conference on Machine Learning (ICML), 2014.
[12] M. W. Mahoney. Randomized algorithms for matrices and data. Foundations and Trends in Machine Learning, 3(2):123–224, 2011.
[13] M. W. Mahoney and P. Drineas. CUR matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences, 106(3):697–702, 2009.
[14] J. A. Tropp. Improved analysis of the subsampled randomized hadamard transform. Advances in Adaptive Data Analysis, 3(01–02):115–126, 2011.
[15] S. Wang and Z. Zhang. Improving CUR matrix decomposition and the Nyström approximation via adaptive sampling. Journal of Machine Learning Research, 14:2729–2769, 2013.
[16] S. Wang and Z. Zhang. Efficient algorithms and error analysis for the modified nyström method. In
International Conference on Artificial Intelligence and Statistics (AISTATS)
, 2014.

Sharpened Error Bounds for Random Sampling Based ℓ_2 Regression

Authors

Time-Space Lower Bounds for Simulating Proof Systems with Quantum and Randomized Verifiers

L1 Regression with Lewis Weights Subsampling

Stability of Sampling for CUR Decompositions

An Explicit Sampling Dependent Spectral Error Bound for Column Subset Selection

Distributed Adaptive Sampling for Kernel Matrix Approximation

Provable Deterministic Leverage Score Sampling

Backfitting for large scale crossed random effects regressions

1 Introduction

1.1 Contributions

2 Preliminaries and Previous Work

3 Main Results

Theorem 3.1 (The Leverage Score Based Sampling)

Theorem 3.2 (Uniform Sampling)

4 Proof

4.1 Key Lemmas

Lemma 1 (Deterministic Error Bound, Lemma 1 and 2 of [6])

Proof

Lemma 2 (Theorem 7 of [5])

Lemma 3 (Theorem 2.2 of [14])

4.2 Proof of Theorem 3.1

Proof

4.3 Proof of Theorem 3.2

Proof

References

Sharpened Error Bounds for Random Sampling Based ℓ_2 Regression

Authors

Related Research

Time-Space Lower Bounds for Simulating Proof Systems with Quantum and Randomized Verifiers

L1 Regression with Lewis Weights Subsampling

Stability of Sampling for CUR Decompositions

An Explicit Sampling Dependent Spectral Error Bound for Column Subset Selection

Distributed Adaptive Sampling for Kernel Matrix Approximation

Provable Deterministic Leverage Score Sampling

Backfitting for large scale crossed random effects regressions

This week in AI

1 Introduction

1.1 Contributions

2 Preliminaries and Previous Work

3 Main Results

Theorem 3.1 (The Leverage Score Based Sampling)

Theorem 3.2 (Uniform Sampling)

4 Proof

4.1 Key Lemmas

Lemma 1 (Deterministic Error Bound, Lemma 1 and 2 of [6])

Proof

Lemma 2 (Theorem 7 of [5])

Lemma 3 (Theorem 2.2 of [14])

4.2 Proof of Theorem 3.1

Proof

4.3 Proof of Theorem 3.2

Proof

References