Optimal choice of k for k-nearest neighbor regression

1 Introduction

1.1 $k$ -NN Algorithm

The $k$ -nearest neighbor algorithm ( $k$ -NN) is a non-parametric method used for classification and regression. For a given sample of $n$ pairs $(x_{i}, y_{i}) \in R^{d} \times R$ and $x \in R^{d}$ the $k$ -NN algorithm outputs

^y = {^m}_{k, n} (x) = \frac{1}{k} \sum j \in N_{k} (x) y_{j},

(1.1)

as an estimate of $m (x) := E [y ∣ x]$ , where $N_{k} (x)$ is the set of indices of the $k$ nearest neighbors of $x$ among $x_{i}$ ’s.

The choice of $k$ is very important. For small values of $k$ , the $k$

-NN estimator would have high variance and may overfit to the noise. As

k

grows, the estimator becomes less flexible and therefore more biased.

1.2 Related Literature

The consistency and asymptotic behaviour of $k$ -NN regression and classification has been studied by many researchers. In [2, 1], the authors provide the necessary and sufficient conditions on $k$ for $k$ -NN estimator to be consistent. In [12], the author has shown asymptotic normality of the $k$ -NN estimator. The rate of convergence of this estimator has also been studied under assumptions on the density of $x_{i}$ ’s [3, 7, 5] or the Lipschitz property of the unknown function $m (\cdot)$ [9].

In applications, an optimal choice of $k$ depends on the sample. Assuming that the variance of noise, $v a r (y ∣ x) = σ^{2}$ , is known and $m (\cdot)$ is Lipschitz with known constant $L$ , Guerre [4] suggested a choice of $k$ as a function of the sample and provided a non-asymptotic bound on the mean squared error of the proposed $k$ -NN estimator conditional on the $x_{i}$ ’s. Although he did not assume independence of the $x_{i}$ ’s, the assumptions on $m (.)$ and $σ^{2}$ seem to be very strong for many real data applications.

In application, a common approach for choosing the value of $k$ is by cross-validation [8]. In [10], Li showed that the $k$ -NN estimator using the $k$ chosen by LOOCV is asymptotically consistent. This result is under the assumption that the distribution of $x_{i}$ ’s fulfills two regularity conditions that together imply that $x_{i}$ ’s are dense in their support in a uniform way. Although this result is stronger than the results on the consistency of the $k$ -NN estimator for non-random choice of $k$ , it does not show why the LOOCV choice of $k$ is a competitive choice. More precisely, given other values of $k$ that give a consistent $k$ -NN estimate, [10] does not show why one should use the value of $k$ by LOOCV.

1.3 Our Work

In this work, we study the LOOCV $k$ -NN estimator. Although it has been shown previously that this $k$ gives us a consistent estimator, it was not shown that this choice of $k$ is optimal. In this paper we compare the mean squared error of the proposed $k$ -NN estimator with the minimum mean squared error using the $k$ -NN algorithm, where the minimum is taken over all choices of $k$ , and we show that with high probability they are very close. In section 2 we discuss the setting and provide the main result. In section 3 we discuss a simulated example. Finally we provide all the proofs in section 4.

1.4 Notation

We use boldface for vectors and matrices, e.g.

X

for a matrix and

x

for a vector. For simplicity, we use

[n]

for

{1, \dots, n}

, the set of natural numbers less than or equal to

n

. For given points

x_{1}, \dots, x_{t}

and

x

, and

k \in N

where

k \leq t

we define

N_{k} (x)

to be the set of indices of the

k

nearest neighbors of

x

among

x_{i}

’s. Ties are broken uniformly at random. For matrix

A

, the

2

-norm

∥ A ∥_{2}

and the Frobenius norm

∥ A ∥_{F}

are defined as

Throughout the paper, $c$ , $C$ and $K$ denote positive absolute constants.

2 Main result

Let ${(x_{i}, y_{i})} \subset R^{d} \times R$ be a set of pairs of observations for $i \in [n]$ . For each $i$ , we assume that $y_{i} = m (x_{i}) + ϵ_{i}$ , where $m : R^{d} \to R$ is an unknown continuous function and $x_{i}$ are drawn independently from an unknown distribution. For simplicity let $μ_{i} = m (x_{i})$ . We assume that $μ_{i} = m (x_{i})$ are sub-Gaussian with sub-Gaussian norm bounded by $C$

E exp (μ_{i} / C)^{2} \leq 2,

and $E [μ_{i}^{2}] \leq M$ .

Noise variables $ϵ_{i}$

’s are independent sub-Gaussian mean zero random variables with sub-Gaussian norm

∥ ϵ_{i} ∥_{ψ^{2}}

upper bounded by

K

E exp (ϵ_{i} / K)^{2} \leq 2.

For $x \in R^{d}$ , the $k$ -NN estimate of $m (x)$ given sample ${(x_{i}, y_{i})}_{i \in [n]}$ is

{^m}_{k, n} (x) = \frac{1}{k} \sum j \in N_{k} (x) y_{j} .

(2.1)

The mean squared error of this estimate is

MSE (k) = E [(m (x) - {^m}_{k, n} (x))^{2}],

(2.2)

where the expectation is with respect to the joint distribution of

x

and

ϵ

In practice, we do not know the probability distribution function of the

x_{i}

’s. Therefore we can not compute the

MSE (k)

. Instead, we can use the given data to estimate the

MSE (k)

For each $i \in [n]$ , let $N_{k} (i) := N_{k} (x_{i})$ be the set of the $k$ nearest neighbors of $x_{i}$ among ${x_{j}}_{j \neq i}$ . Note that in defining $N_{k} (i)$ we are excluding $x_{i}$ from the set of query points.

Define

k^{*} := {argmin}_{k \in {1, \dots, n - 1}} MSE (k) .

(2.3)

One may find $k^{*}$ the best value of $k$ for the $k$ -NN estimate. But since the distributions of $x$ and $ϵ$ and function $m (\cdot)$ are all unknown, in practice we can not find $k^{*}$ . Instead, for each $k \in [n - 1]$ we define

f (k) := \frac{1}{n} n \sum i = 1 (y_{i} - \frac{1}{k} \sum j \in N_{k} (i) y_{j})^{2},

(2.4)

and

~ k := {argmin}_{k \in {1, \dots, n - 1}} f (k) .

(2.5)

In the Statistics and Machine Learning literature

f (k)

is known as the leave-one-out cross-validation (LOOCV) estimate of the mean squared error. Note that

f : [n - 1] \to R

is a random function (randomness comes from the dependence of

f (k)

ϵ_{i}

’s) and therefore

~ k

is a random variable. For each given sample

{(x_{i}, y_{i})}_{i \in [n]}

, we can compute

~ k

. Therefore a simple idea is to use

~ k

in the

k

-NN algorithm. Note that in practice the distance between

~ k

and

k^{*}

is not of the main importance for us. The main question is how far

MSE (~ k)

is from

MSE (k^{*})

? Theorem 2.1 gives a probability tail bound on

| MSE (k^{*}) - MSE (~ k) |

to answer this question.

Theorem 2.1.

With $k^{*}$ and $~ k$ defined in 2.3 and 2.5,

	$P(\|\emphMSE(k∗)−\emphMSE(~k)\|≥t)$

			$\leq 2 (n + 1) exp [- n c min (\frac{t^{2}}{32 K^{4} (1 + 4 γ_{d})}, \frac{t}{16 K^{2} (1 + γ_{d})^{2}})]$
			$+ 2 (n + 1) exp (- \frac{n t^{2}}{2048 (1 + γ_{d})^{4} (M + t)})$
			$+ (n + 1) exp (- \frac{2 n t^{2}}{C}),$

where $γ_{d}$ is a constant that only depends on $d$ . Constants $K$ , and $C$ are upper bounds on the sub-Gaussian norms of $ϵ_{i}$ and $μ_{i}$ respectively, and $E [μ_{i}^{2}] \leq M$ .

For the set of observations ${(x_{i}, y_{i})}_{i \in [n]}$ from an unknown joint distribution, the choice of $k^{*}$ for the $k$ -NN estimate gives us the minimum mean squared error over all possible choices of $k$ , which is typically not computable. Instead, Theorem 2.1 guarantees that using $~ k$ , with high probability, gives $MSE (~ k)$ that is very close to $MSE (k^{*})$ . This shows that not only does $~ k$ give us a consistent estimator but it is an optimal choice as well.

3 Discussion and Simulations

We should emphasize that in computing $~ k$ we exclude each $x_{i}$ from the whole set and we do not consider $x_{i}$ as one of its nearest neighbors. This is in fact very important and prevents us from choosing a value of $k$ that suffers from overfitting. The following example helps to see this better.

Example 3.1.

For n = 1000 we have generated $x_{i}$ ’s i.i.d from $Unif [0, 1]$ and $ϵ_{i}$ ’s i.i.d from $N (0, 1)$ . Let $m (x) = cos (20 x) + x / 2$ and $y_{i} = m (x_{i}) + ϵ_{i}$ . For this sample we have $k^{*} = 58$ and $~ k = 48$ . Now let $N_{k}^{†} (x_{i})$ be the set of $k$ nearest neighbors of $x_{i}$ among all $x_{1}, \dots, x_{n}$ and define

k^{†} := {argmin}_{k \in {2, \dots, n}} \sum i \in [n] (y_{i} - \frac{1}{k} \sum j \in N_{k}^{†} (x_{i}) y_{j})^{2} .

(3.1)

Note that we are taking the minimum over $k \geq 2$ , since clearly for $k = 1$ the sum in the right hand side of 3.1 is equal to zero. For our sample $ˆ MSE (k^{*}) = 0.010$ , $ˆ MSE (~ k) = 0.013$ and $ˆ MSE (k^{†}) = 0.500$ , where $ˆ MSE$ is the empirical estimate of MSE,

ˆ MSE (k) = \frac{1}{n} n \sum i = 1 (m (x_{i}) - {^m}_{k, n} (x_{i}))^{2} .

It is clear that by choosing $k^{†} = 2$ , we will overfit to the noise and therefore the estimated mean squared error is much higher than that of two other values of $k$ .

Figure 1: Plot of $f (k)$ (solid line) and $ˆ MSE (k)$ (dashed line) for $k \in [n - 1]$ .

In Figure 1 we have plotted the $f (k)$ (solid line) and $ˆ MSE (k)$ (dashed line) for $k \in [n - 1]$ . It can be seen that the behaviour of these two is very similar. In fact, as we expect from equality 4.1 in Section 4,

E [f (k)] = \frac{1}{n} n \sum i = 1 E [ϵ_{i}^{2}] + MSE (k),

they differ slightly only by a constant. Therefore the point-wise difference of these two curves, when $ϵ_{i}$ ’s have the same variance $σ^{2}$ is approximately equal to $σ^{2}$ . This shows that looking at the curve of $f (k)$ gives us almost the same information as looking at the curve $MSE (k)$ . Therefore computing $f (k)$ is enough for finding the optimal choice of $k$ .

(a) Plot of $m (x_{i})$ ’s (solid) and $y_{i}$ ’s (dashed).

From Figure 1(a) and 1(b) it can be seen that $k$ -NN has almost recovered the signal in the presence of the strong noise.

4 proofs

Proof.

Theorem 2.1.

By writing $y_{i} = μ_{i} + ϵ_{i}$ , we have

$f (k)$	$=$	$\frac{1}{n} n \sum i = 1 (y_{i} - \frac{1}{k} \sum j \in N_{k} (i) y_{j})^{2}$
	$=$	$\frac{1}{n} n \sum i = 1 (μ_{i} + ϵ_{i} - \frac{1}{k} \sum j \in N_{k} (i) y_{j})^{2}$
	$=$	$\frac{1}{n} n \sum i = 1 (μ_{i} - \frac{1}{k} \sum j \in N_{k} (i) y_{j})^{2} + ϵ_{i}^{2} + 2 ϵ_{i} (μ_{i} - \frac{1}{k} \sum j \in N_{k} (i) y_{j}) .$

Since $ϵ_{i}$ ’s are independent with mean zero, taking expectation of the above equality gives us

E [f (k)] = \frac{1}{n} n \sum i = 1 E [ϵ_{i}^{2}] + MSE (k) .

(4.1)

Define $g (k) := E [f (k)]$ , for $k \in [n - 1]$ . For each $n$ , $g : [n - 1] \to R$ is a deterministic function and does not depend on a given sample. Remember that $~ k$ is a function of the given sample and therefore is random. So $g (~ k)$ depends on the given sample as well and therefore is random. By definition of $k^{*}$ , for all $k \in [n - 1]$ , $MSE (k^{*}) \leq MSE (k)$ and therefore $MSE (k^{*}) \leq MSE (~ k)$ This gives us $g (k^{*}) \leq g (~ k)$ . Also by definition of $~ k$ , we have $f (~ k) \leq f (k^{*})$ . Putting these two together gives us,

$P (\| MSE (k^{*}) - MSE (~ k) \| \geq t)$		(4.2)

	$= P (\| g (k^{*}) - g (~ k) \| \geq t)$
	$\leq P (\| g (k^{}) - f (k^{}) \| \geq t / 2) + P (\| g (~ k) - f (~ k) \| \geq t / 2)$
	$= P (\| E [f (k^{})] - f (k^{}) \| \geq t / 2) + P (\| E [f (~ k)] - f (~ k) \| \geq t / 2) .$

Therefore to find an upper bound on $P (| MSE (k^{*}) - MSE (~ k) | \geq t)$ , it’s enough to find an upper bound on $P (| E [f (k)] - f (k) | \geq t)$ for any arbitrary $k \in [n - 1]$ .

Lemma 4.1.

For any $k \in [n - 1]$ and any $λ > 0$

$P (\| f (k) - E [f (k)] \| \geq t)$	$\leq$	$2 exp [- n c min (\frac{t^{2}}{8 K^{4} (1 + 4 γ_{d})}, \frac{t}{8 K^{2} (1 + γ_{d})^{2}})]$
	$+$	$2 exp (- \frac{n t^{2}}{512 (1 + γ_{d})^{4} (M + λ)})$
	$+$	$exp (- \frac{2 n λ^{2}}{C}),$

where $γ_{d} > 0$ is a constant that only depends on the dimension $d$ , and $K$ and $C$ are upper bounds on the sub-Gaussian norm of $ϵ_{i}$ and $μ_{i}$ , and $E [μ_{i}^{2}] \leq M$ .

Proof.

Define the nonsymmetric $n \times n$ matrix $B = [b_{i j}]$ in the following way,

b_{i j} := ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} 1 & i = j 0 & j \notin N_{k} (i) - 1 / k & j \in N_{k} (j) \end{matrix} .

(4.3)

Let $A = [a_{i j}] = B^{T} B / n$ . Also let $ϵ = (ϵ_{1}, \dots, ϵ_{n})$ and $μ = (μ_{1}, \dots, μ_{n})$ . Then we can rewrite $f (k)$ in the following vector product form

f (k) = (ϵ + μ)^{T} A (ϵ + μ) .

(4.4)

Using the triangle inequality, we have

	$\| f (k) - E [f (k)] \|$	$=$	$\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] + 2 ϵ^{T} A μ \|$
		$\leq$	$\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] \| + 2 \| ϵ^{T} A μ \| .$

Therefore it’s enough to find probability tail bounds on $| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] |$ and $| ϵ^{T} A μ |$ . Note that $A$ is random. To find such bounds we need to have information on the norm of $A$ . Lemmas 4.2 and 4.3 provide uniform bound on $∥ A ∥_{2}$ and $∥ A ∥_{F}$ .

Lemma 4.2.

For $A = B^{T} B / n$ , where entries of $B$ are defined in 4.3, we have

∥ A ∥_{F}^{2} \leq \frac{2 (1 + 4 γ_{d})}{n},

(4.5)

where $γ_{d}$ is a constant that only depends on $d$ .

Lemma 4.3.

For $A = B^{T} B / n$ , where entries of $B$ are defined in 4.3, we have

∥ A ∥_{2} \leq \frac{4}{n} (1 + γ_{d})^{2},

(4.6)

where $γ_{d}$ is a constant that only depends on $d$ .

Bound on $| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] |$ . Conditional on given sample ${x_{i}}_{i \in [n]}$ and therefore given $A$ , by Hanson-Wright inequality [11] and Lemmas 4.2 and 4.3 we have

$P (| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}] | \geq t ∣ {x_{i}}_{i \in [n]})$ (4.7)

$\leq 2 exp ⎡ ⎣ - c min ⎛ ⎝ \frac{t^{2}}{K^{4} ∥ A ∥_{F}^{2}}, \frac{t}{K^{2} ∥ A ∥_{2}} ⎞ ⎠ ⎤ ⎦$

$\leq 2 exp [- n c min (\frac{t^{2}}{2 K^{4} (1 + 4 γ_{d})}, \frac{t}{4 K^{2} (1 + γ_{d})^{2}})] .$

Also note that

$E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}]$ $=$ $\frac{1}{n} E [⟨ ϵ^{T} B, ϵ^{T} B ⟩]$ (4.8)

$=$ $\frac{1}{n} E [⟨ n \sum i = 1 ϵ_{i} B_{i}, n \sum i = 1 ϵ_{i} B_{i} ⟩]$

$=$ $\frac{(1 + 1 / k)}{n} n \sum i = 1 E [ϵ_{i}^{2}]$

$=$ $(1 + \frac{1}{k}) \frac{∥ ϵ ∥_{2}^{2}}{n} .$

The right side of 4.8 does not depend on sample ${x_{i}}_{i \in [n]}$ . Therefore $E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}]$ is almost surely constant,

$E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}] = E [ϵ^{T} A ϵ] .$ (4.9)

Putting 4.9 and 4.7 together gives us

$P (| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] | \geq t ∣ {x_{i}}_{i \in [n]})$ (4.10)

$= P (| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}] | \geq t ∣ {x_{i}}_{i \in [n]})$

$\leq 2 exp [- n c min (\frac{t^{2}}{2 K^{4} (1 + 4 γ_{d})}, \frac{t}{4 K^{2} (1 + γ_{d})^{2}})] .$

Note that 4.10 does not depend on ${x_{i}}_{i \in [n]}$ , therefore

$P (| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] | \geq t)$ (4.11)

$\leq 2 exp [- n c min (\frac{t^{2}}{2 K^{4} (1 + 4 γ_{d})}, \frac{t}{4 K^{2} (1 + γ_{d})^{2}})] .$
Bound on $| ϵ^{T} A μ |$ .

By the Hoeffding’s inequality for any $t \geq 0$ ,

$P (| ϵ^{T} A μ | \geq t ∣ {x_{i}}_{i \in [n]})$ $=$ $P (| n \sum j = 1 (n \sum i = 1 a_{j i} μ_{i}) ϵ_{j} | \geq t ∣ {x_{i}}_{i \in [n]})$ (4.12)

$\leq$ $2 exp (- \frac{t^{2}}{2 \sum_{j = 1}^{n} (\sum_{i = 1}^{n} a_{j i} μ_{i})^{2}}) .$

Note that

$n \sum j = 1 (n \sum i = 1 a_{j i} μ_{j})^{2}$ $\leq$ $∥ A ∥_{2}^{2} ∥ μ ∥_{2}^{2} .$ (4.13)

Inequalities 4.12 and 4.13 together give

$P (| ϵ^{T} A μ | \geq t ∣ {x_{i}}_{i \in [n]})$ $\leq$ $2 exp (- \frac{t^{2}}{2 ∥ A ∥_{2}^{2} ∥ μ ∥_{2}^{2}}) .$ (4.14)

Now by Lemma 4.3

$P (| ϵ^{T} A μ | \geq t ∣ {x_{i}}_{i \in [n]})$ $\leq$ $2 exp (- \frac{n^{2} t^{2}}{32 (1 + γ_{d})^{4} ∥ μ ∥_{2}^{2}})$

$=$ $2 exp (- \frac{n t^{2}}{32 (1 + γ_{d})^{4} ∥ μ ∥_{2}^{2} / n}) .$

By SLLN, $∥ μ ∥_{2}^{2} / n \to E [μ_{i}^{2}] = M$ almost surely. For any $λ > 0$ , Hoeffding’s inequality gives

$P (∥ μ ∥_{2}^{2} / n \geq E [m (x)^{2}] + λ) \leq exp (- \frac{2 n λ^{2}}{C}) .$

Therefore

$P (| ϵ^{T} A μ | \geq t)$ (4.15)

$= E [P (| ϵ^{T} A μ | \geq t ∣ {x_{i}}_{i \in [n]})]$

$\leq 2 exp (- \frac{n t^{2}}{32 (1 + γ_{d})^{4} (M + λ)}) + exp (- \frac{2 n λ^{2}}{C}) .$

Combining 4.11 and 4.15 gives

$P (\| f (k) - E [f (k)] \| \geq t)$		(4.17)

	$\leq P (\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] \| \geq t / 2) + P (\| ϵ^{T} A μ \| \geq t / 4)$
	$\leq 2 exp [- n c min (\frac{t^{2}}{8 K^{4} (1 + 4 γ_{d})}, \frac{t}{8 K^{2} (1 + γ_{d})^{2}})]$
	$+ 2 exp (- \frac{n t^{2}}{512 (1 + γ_{d})^{4} (M + λ)})$
	$+ exp (- \frac{2 n λ^{2}}{C}) .$

∎

Using Lemma 4.1 for $f (k^{*})$ and $f (~ k)$ and union bound we have

	$P (\| MSE (k^{*}) - MSE (~ k) \| \geq t)$

			$\leq P (\| f (~ k) - E [f (~ k)] \| \geq t / 2) + P (\| f (k^{}) - E [f (k^{})] \| \geq t / 2)$
			$\leq \cup_{k = 1}^{n} P (\| f (k) - E [f (k)] \| \geq t / 2) + P (\| f (k^{}) - E [f (k^{})] \| \geq t / 2)$
			$\leq 2 (n + 1) exp [- n c min (\frac{t^{2}}{32 K^{4} (1 + 4 γ_{d})}, \frac{t}{16 K^{2} (1 + γ_{d})^{2}})]$
			$+ 2 (n + 1) exp (- \frac{n t^{2}}{2048 (1 + γ_{d})^{4} (M + λ)})$
			$+ (n + 1) exp (- \frac{2 n λ^{2}}{C}) .$

Note that $λ > 0$ is arbitrary therefore we simply set $λ = t$ and this completes the proof of Theorem 2.1,

	$P (\| MSE (k^{*}) - MSE (~ k) \| \geq t)$

			$\leq 2 (n + 1) exp [- n c min (\frac{t^{2}}{32 K^{4} (1 + 4 γ_{d})}, \frac{t}{16 K^{2} (1 + γ_{d})^{2}})]$
			$+ 2 (n + 1) exp (- \frac{n t^{2}}{2048 (1 + γ_{d})^{4} (M + t)})$
			$+ (n + 1) exp (- \frac{2 n t^{2}}{C}) .$

∎

4.1 Proof of Lemmas 4.2 and 4.3

Proof.

Lemma 4.2. Let $b_{i}$ be the $i$ -th row of matrix $B$ . Then

$∥ A ∥_{F}^{2}$	$=$	$\frac{1}{n^{2}} Tr (A^{T} A)$
	$=$	$\frac{1}{n^{2}} Tr (B^{T} B B^{T} B)$
	$=$	$\frac{1}{n^{2}} Tr (B B^{T} B B^{T})$
	$=$	$\frac{1}{n^{2}} \sum i, j ⟨ b_{i}, b_{j} ⟩^{2}$
	$=$	$\frac{1}{n} (1 + \frac{1}{k}) + \frac{1}{n^{2}} \sum i \neq j ⟨ b_{i}, b_{j} ⟩^{2}$
	$=$	$\frac{1}{n} (1 + \frac{1}{k}) + \frac{1}{n^{2}} n \sum i = 1 \sum j \neq i ⟨ b_{i}, b_{j} ⟩^{2} .$

For each $i, j \in [n]$ ,

⟨ b_{i}, b_{j} ⟩ = (\frac{- 1}{k}) [1 {i \in N_{k} (j)} + 1 {j \in N_{k} (i)}] + \frac{1}{k^{2}} | N_{k} (i) \cap N_{k} (j) |,

and therefore

| ⟨ b_{i}, b_{j} ⟩ |

\leq

\frac{2}{k} .

(4.18)

This gives

\sum j \neq i ⟨ b_{i}, b_{j} ⟩^{2} \leq \frac{4}{k^{2}} | {j : ⟨ b_{i}, b_{j} ⟩ \neq 0} | .

(4.19)

Now note that for each $i \in [n]$ , $| {j : ⟨ b_{i}, b_{j} ⟩ \neq 0} |$ can not be large. In fact $⟨ b_{i}, b_{j} ⟩ \neq 0$ at most for those $j$ ’s that

(4.20)

By definition $| N_{k} (i) | = k$ and for any $ℓ \in [n]$ by Corollary 6.1. in [6] there are at most $γ_{d} k$ indices $j$ such that $ℓ \in N_{k} (j)$ , where $γ_{d}$ is a constant depends only on $d$ . Therefore

| {j : ⟨ b_{i}, b_{j} ⟩ \neq 0} | \leq γ_{d} k (k + 1) .

(4.21)

This gives us

	$∥ A ∥_{F}^{2}$	$\leq$	$(1 + 4 γ_{d}) (1 + \frac{1}{k}) \frac{1}{n}$
		$\leq$	$\frac{2 (1 + 4 γ_{d})}{n} .$

∎

Proof.

Lemma 4.3. Note that

∥ B ∥_{2} = max ∥ x ∥_{2} = 1 ∥ B x ∥_{2} .

(4.22)

Therefore for any arbitrary $x$ such that $∥ x ∥_{2} = 1$ ,

$∥ B x ∥_{2}$	$=$	$n \sum i = 1 ⟨ b_{i}, x ⟩^{2}$	(4.23)
	$\leq$	$2 n \sum i = 1 x_{i}^{2} + \frac{2}{k^{2}} n \sum i = 1 \sum j \in N_{k} (i) x_{j}^{2}$	(4.24)
	$=$	$2 ∥ x ∥_{2} + \frac{2}{k^{2}} n \sum j = 1 \sum {i : j \in N_{k} (i)} x_{j}^{2}$	(4.25)
	$\leq$	$2 (1 + \frac{γ_{d}}{k^{2}}) ∥ x ∥_{2} .$	(4.26)

Therefore

$∥ A ∥_{2}$	$=$	$\frac{∥ B ∥_{2}^{2}}{n}$
	$\leq$	$\frac{4}{n} (1 + \frac{γ_{d}}{k^{2}})^{2}$
	$\leq$	$\frac{4}{n} (1 + γ_{d})^{2} .$

∎

An R language package knnopt will soon be made available on the CRAN repository.

Acknowledgement

The author is very grateful to her advisor Sourav Chatterjee for his constant encouragement and insightful conversations and comments.

References

Devroye [1994] Devroye, L; Györfi, L; Krzyzak, A and Lugosi, G.(1994). On the Strong Universal Consistency of Nearest Neighbor Regression Function Estimates. The Annals of Statistics, 22 no. 3 (1994), 1371–1385. 1311980
Devroye [1982] Devroye, L (1982). Necessary and Sufficient Conditions For The Pointwise Convergence of Nearest Neighbor Regression Function Estimates. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete., 61 no. 4 (1982), 467–481. 0682574
Fan [1993] Fan, J.
(1993) Local Linear Regression Smoothers and Their Minimax Efficiencies.
Ann. Statist. 21 no. 1, 196–216. 1212173
Guerre [2000] Guerre, E. (2000) Design Adaptive Nearest Neighbor Regression Estimation. J. Multivariate Anal. 75 no. 2, 219–244. 1802549
Györfi [1981] Györfi, K. C. (1981) The Rate of Convergence of $k_{n}$ -NN Regression Estimates and Classification Rules. IEEE Trans. Inform. Theory. 27 no. 3, 362–364.
Györfi, Kohler, Krzyźak and Walk [2002] Györfi, L., Kohler, M., Krzyźak, A. and Walk, H. (2002). A Distribution-Free Theory of Non-parametric Regression. Springer.
Hall, Marron and Neumann [1997] Hall, P. Marron, J. S. Neumann, M. H. and Titterington, D. M. (1997) Curve Estimation When The Design Density Is Low. Ann. Statist. 25 no. 2, 756–770. 1439322
James [2013] James, G. Witten, D. Hastie, T. and Tibshirani, R. (2013) An Introduction to Statistical Learning : with Applications in R. Springer.
Kulkarni and Ponser [1995] Kulkarni, S.R. and Ponser, S.E. (1995) Rates of Convergence of Nearest Neighbor Estimation Under Arbitrary Sampling. IEEE Trans. Inform. Theory. 41 no. 4, 1028–1039. 1366756
Li [1984] Li, K. C. (1984) Consistency For Cross-validated Nearest Neighbor Estimates in Non-parametric Regression. Ann. Statist. 12 no. 1, 230–240. 0733510
Rudelson and Vershynin [2013] Rudelson, M. and Vershynin, R. (2013). Hanson-Wright Inequality and sub-Gaussian Concentration. Electron. Commun. Probab., 18(2013) 3125258
Stute [1984] Stute, W.(1994). Asymptotic Normality of Nearest Neighbor Regression Function Estimates. The Annals of Statistics, 12 no. 3 (1984), 917–929. 1397508

$P (\| MSE (k^{*}) - MSE (~ k) \| \geq t)$		(4.2)

	$= P (\| g (k^{*}) - g (~ k) \| \geq t)$
	$\leq P (\| g (k^{}) - f (k^{}) \| \geq t / 2) + P (\| g (~ k) - f (~ k) \| \geq t / 2)$
	$= P (\| E [f (k^{})] - f (k^{}) \| \geq t / 2) + P (\| E [f (~ k)] - f (~ k) \| \geq t / 2) .$

	$\| f (k) - E [f (k)] \|$	$=$	$\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] + 2 ϵ^{T} A μ \|$
		$\leq$	$\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] \| + 2 \| ϵ^{T} A μ \| .$

Optimal choice of k for k-nearest neighbor regression

How many samples are needed to reliably approximate the best linear estimator for a linear inverse problem?

Metric Effects based on Fluctuations in values of k in Nearest Neighbor Regressor

Generalized Linear Tree Space Nearest Neighbor

Non-separable Nearest-Neighbor Gaussian Process Model for Antarctic Surface Mass Balance and Ice Core Site Selection

Modeling Daily Pan Evaporation in Humid Climates Using Gaussian Process Regression

The Theory of Perfect Learning

Bayesian Model Selection Methods for Mutual and Symmetric k-Nearest Neighbor Classification

1 Introduction

1.1 $k$ -NN Algorithm

1.2 Related Literature

1.3 Our Work

1.4 Notation

2 Main result

Theorem 2.1.

3 Discussion and Simulations

Example 3.1.

4 proofs

Proof.

Lemma 4.1.

Proof.

Lemma 4.2.

Lemma 4.3.

4.1 Proof of Lemmas 4.2 and 4.3

Proof.

Proof.

Acknowledgement

References

$P (\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}] \| \geq t ∣ {x_{i}}_{i \in [n]})$		(4.7)

	$\leq 2 exp ⎡ ⎣ - c min ⎛ ⎝ \frac{t^{2}}{K^{4} ∥ A ∥_{F}^{2}}, \frac{t}{K^{2} ∥ A ∥_{2}} ⎞ ⎠ ⎤ ⎦$
	$\leq 2 exp [- n c min (\frac{t^{2}}{2 K^{4} (1 + 4 γ_{d})}, \frac{t}{4 K^{2} (1 + γ_{d})^{2}})] .$

$E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}]$	$=$	$\frac{1}{n} E [⟨ ϵ^{T} B, ϵ^{T} B ⟩]$	(4.8)
	$=$	$\frac{1}{n} E [⟨ n \sum i = 1 ϵ_{i} B_{i}, n \sum i = 1 ϵ_{i} B_{i} ⟩]$
	$=$	$\frac{(1 + 1 / k)}{n} n \sum i = 1 E [ϵ_{i}^{2}]$
	$=$	$(1 + \frac{1}{k}) \frac{∥ ϵ ∥_{2}^{2}}{n} .$

$P (\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] \| \geq t ∣ {x_{i}}_{i \in [n]})$		(4.10)

	$= P (\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ ∣ {x_{i}}_{i \in [n]}] \| \geq t ∣ {x_{i}}_{i \in [n]})$
	$\leq 2 exp [- n c min (\frac{t^{2}}{2 K^{4} (1 + 4 γ_{d})}, \frac{t}{4 K^{2} (1 + γ_{d})^{2}})] .$

$P (\| ϵ^{T} A ϵ - E [ϵ^{T} A ϵ] \| \geq t)$		(4.11)

	$\leq 2 exp [- n c min (\frac{t^{2}}{2 K^{4} (1 + 4 γ_{d})}, \frac{t}{4 K^{2} (1 + γ_{d})^{2}})] .$

	$P (\| ϵ^{T} A μ \| \geq t ∣ {x_{i}}_{i \in [n]})$	$=$	$P (\| n \sum j = 1 (n \sum i = 1 a_{j i} μ_{i}) ϵ_{j} \| \geq t ∣ {x_{i}}_{i \in [n]})$		(4.12)
		$\leq$	$2 exp (- \frac{t^{2}}{2 \sum_{j = 1}^{n} (\sum_{i = 1}^{n} a_{j i} μ_{i})^{2}}) .$		(4.12)

	$P (\| ϵ^{T} A μ \| \geq t ∣ {x_{i}}_{i \in [n]})$	$\leq$	$2 exp (- \frac{n^{2} t^{2}}{32 (1 + γ_{d})^{4} ∥ μ ∥_{2}^{2}})$
		$=$	$2 exp (- \frac{n t^{2}}{32 (1 + γ_{d})^{4} ∥ μ ∥_{2}^{2} / n}) .$

$P (\| ϵ^{T} A μ \| \geq t)$		(4.15)

	$= E [P (\| ϵ^{T} A μ \| \geq t ∣ {x_{i}}_{i \in [n]})]$
	$\leq 2 exp (- \frac{n t^{2}}{32 (1 + γ_{d})^{4} (M + λ)}) + exp (- \frac{2 n λ^{2}}{C}) .$

Optimal choice of k for k-nearest neighbor regression

Related Research

How many samples are needed to reliably approximate the best linear estimator for a linear inverse problem?

Metric Effects based on Fluctuations in values of k in Nearest Neighbor Regressor

Generalized Linear Tree Space Nearest Neighbor

Non-separable Nearest-Neighbor Gaussian Process Model for Antarctic Surface Mass Balance and Ice Core Site Selection

Modeling Daily Pan Evaporation in Humid Climates Using Gaussian Process Regression

The Theory of Perfect Learning

Bayesian Model Selection Methods for Mutual and Symmetric k-Nearest Neighbor Classification

1 Introduction

1.1 k-NN Algorithm

1.2 Related Literature

1.3 Our Work

1.4 Notation

2 Main result

Theorem 2.1.

3 Discussion and Simulations

Example 3.1.

4 proofs

Proof.

Lemma 4.1.

Proof.

Lemma 4.2.

Lemma 4.3.

4.1 Proof of Lemmas 4.2 and 4.3

Proof.

Proof.

Acknowledgement

References

1.1 $k$ -NN Algorithm