The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal

1 Introduction

Information theoretic measures such as entropy, Kullback-Leibler divergence and mutual information quantify the amount of information among random variables. They have many applications in modern machine learning tasks, such as classification

[48], clustering [46, 58, 10, 41]

and feature selection

[1, 17]

. Information theoretic measures and their variants can also be applied in several data science domains such as causal inference

[18], sociology [49] and computational biology [36]

. Estimating information theoretic measures from data is a crucial sub-routine in the aforementioned applications and has attracted much interest in statistics community. In this paper, we study the problem of estimating Shannon differential entropy, which is the basis of estimating other information theoretic measures for continuous random variables.

Suppose we observe $n$

independent identically distributed random vectors

X = {X_{1}, \dots, X_{n}}

drawn from density function

f

where

X_{i} \in R^{d}

. We consider the problem of estimating the differential entropy

h (f)

= - \int f (x) ln f (x) d x,

(1)

from the empirical observations $X$ . The fundamental limit of estimating the differential entropy is given by the minimax risk

inf^h sup f \in F {(E (^h (X) - h (f))^{2})}^{1 / 2},

(2)

where the infimum is taken over all estimators $^h$ that is a function of the empirical data $X$ . Here $F$ denotes a (nonparametric) class of density functions.

The problem of differential entropy estimation has been investigated extensively in the literature. As discussed in [2]

, there exist two main approaches, where one is based on kernel density estimators

[30], and the other is based on the nearest neighbor methods [56, 53, 52, 11, 3], which is pioneered by the work of [33].

The problem of differential entropy estimation lies in the general problem of estimating nonparametric functionals. Unlike the parametric counterparts, the problem of estimating nonparametric functionals is challenging even for smooth functionals. Initial efforts have focused on inference of linear, quadratic, and cubic functionals in Gaussian white noise and density models and have laid the foundation for the ensuing research. We do not attempt to survey the extensive literature in this area, but instead refer to the interested reader to, e.g.,

[24, 5, 12, 16, 6, 32, 37, 47, 8, 9, 54] and the references therein. For non-smooth functionals such as entropy, there is some recent progress [38, 26, 27] on designing theoretically minimax optimal estimators, while these estimators typically require the knowledge of the smoothness parameters, and the practical performances of these estimators are not yet known.

The $k$ -nearest neighbor differential entropy estimator, or Kozachenko-Leonenko (KL) estimator is computed in the following way. Let $R_{i, k}$ be the distance between $X_{i}$ and its $k$ -nearest neighbor among ${X_{1}, \dots, X_{i - 1}, X_{i + 1}, \dots, X_{n}}$ . Precisely, $R_{i, k}$ equals the $k$ -th smallest number in the list ${∥ X_{i} - X_{j} ∥ : j \neq i, j \in [n]}$ , here $[n] = {1, 2, \dots, n}$ . Let $B (x, ρ)$ denote the closed $ℓ_{2}$ ball centered at $x$ of radius $ρ$ and $λ$ be the Lebesgue measure on $R^{d}$ . The KL differential entropy estimator is defined as

{^h}_{n, k} (X)

= ln k - ψ (k) + \frac{1}{n} n \sum i = 1 ln (\frac{n}{k} λ (B (X_{i}, R_{i, k}))),

(3)

where $ψ (x)$ is the digamma function with $ψ (1) = - γ$ , $γ = - \int_{0}^{\infty} e^{- t} ln t d t = 0.5772156 \dots$ is the Euler–Mascheroni constant.

There exists an intuitive explanation behind the construction of the KL differential entropy estimator. Writing informally, we have

h (f) = E_{f} [- ln f (X)] \approx \frac{1}{n} n \sum i = 1 - ln f (X_{i}) \approx \frac{1}{n} n \sum i = 1 - ln^f (X_{i}),

(4)

where the first approximation is based on the law of large numbers, and in the second approximation we have replaced

f

by a nearest neighbor density estimator

^f

. The nearest neighbor density estimator

^f (X_{i})

follows from the “intuition” ¹¹1Precisely, we have

\int_{B (X_{i}, R_{i, k})} f (u) d u \sim B e t a (k, n - k)

[4, Chap. 1.2]. A

B e t a (k, n - k)

distributed random variable has mean

\frac{k}{n}

. that

^f (X_{i}) λ (B (X_{i}, R_{i, k})) \approx \frac{k}{n} .

(5)

Here the final additive bias correction term $ln k - ψ (k)$ follows from a detailed analysis of the bias of the KL estimator, which will become apparent later.

We focus on the regime where $k$ is a fixed: in other words, it does not grow as the number of samples $n$ increases. The fixed $k$ version of the KL estimator is widely applied in practice and enjoys smaller computational complexity, see [52].

There exists extensive literature on the analysis of the KL differential entropy estimator, which we refer to [4] for a recent survey. One of the major difficulties in analyzing the KL estimator is that the nearest neighbor density estimator exhibits a huge bias when the density is small. Indeed, it was shown in [42] that the bias of the nearest neighbor density estimator in fact does not vanish even when $n \to \infty$ and deteriorates as $f (x)$ gets close to zero. In the literature, a large collection of work assume that the density is uniformly bounded away from zero [23, 29, 57, 30, 53], while others put various assumptions quantifying on average how close the density is to zero [25, 40, 56, 14, 20, 52, 11]. In this paper, we focus on removing assumptions on how close the density is to zero.

1.1 Main Contribution

Let $H_{d}^{s} (L; [0, 1]^{d})$ be the Hölder ball in the unit cube (torus) (formally defined later in Definition 2 in Appendix A) and $s \in (0, 2]$ is the Hölder smoothness parameter. Then, the worst case risk of the fixed $k$ -nearest neighbor differential entropy estimator over $H_{d}^{s} (L; [0, 1]^{d})$ is controlled by the following theorem.

Theorem 1

Let $X = {X_{1}, \dots, X_{n}}$ be i.i.d. samples from density function $f$ . Then, for $0 < s \leq 2$ , the fixed $k$ -nearest neighbor KL differential entropy estimator ${^h}_{n, k}$ in (3) satisfies

{⎛ ⎝ sup f \in H_{d}^{s} (L; [0, 1]^{d}) E_{f} {({^h}_{n, k} (X) - h (f))}_{n, k}^{2} ⎞ ⎠}_{f}^{\frac{1}{2}} \leq C (n^{- \frac{s}{s + d}} ln (n + 1) + n^{- \frac{1}{2}}) .

(6)

where $C$ is a constant depends only on $s, L, k$ and $d$ .

The KL estimator is in fact nearly minimax up to logarithmic factors, as shown in the following result from [26].

Theorem 2

[26] Let $X = {X_{1}, \dots, X_{n}}$ be i.i.d. samples from density function $f$ . Then, there exists a constant $L_{0}$ depending on $s, d$ only such that for all $L \geq L_{0}, s > 0$ ,

{⎛ ⎝ inf^h sup f \in H_{d}^{s} (L; [0, 1]^{d}) E_{f} {(^h (X) - h (f))}^{2} ⎞ ⎠}_{f}^{\frac{1}{2}} \geq c (n^{- \frac{s}{s + d}} (ln (n + 1))^{- \frac{s + 2 d}{s + d}} + n^{- \frac{1}{2}}) .

(7)

where $c$ is a constant depends only on $s$ , $L$ and $d$ .

Remark 1

We emphasize that one cannot remove the condition $L \geq L_{0}$ in Theorem 2. Indeed, if the Hölder ball has a too small width, then the density itself is bounded away from zero, which makes the differential entropy a smooth functional, with minimax rates $n^{- \frac{4 s}{4 s + d}} + n^{- 1 / 2}$ [51, 50, 43].

Theorem 1 and 2 imply that for any fixed $k$ , the KL estimator achieves the minimax rates up to logarithmic factors without knowing $s$ for all $s \in (0, 2]$ , which implies that it is near minimax rate-optimal (within logarithmic factors) when the dimension $d \leq 2$ . We cannot expect the vanilla version of the KL estimator to adapt to higher order of smoothness since the nearest neighbor density estimator can be viewed as a variable width kernel density estimator with the box kernel, and it is well known in the literature (see, e.g., [55, Chapter 1]) that any positive kernel cannot exploit the smoothness $s > 2$ . We refer to [26] for a more detailed discussion on this difficulty and potential solutions. The Jackknife idea, such as the one presented in [11, 3] might be useful for adapting to $s > 2$ .

The significance of our work is multi-folded:

We obtain the first uniform upper bound on the performance of the fixed $k$ -nearest neighbor KL differential entropy estimator over Hölder balls without assuming how close the density could be from zero. We emphasize that assuming conditions of this type, such as the density is bounded away from zero, could make the problem significantly easier. For example, if the density $f$ is assumed to satisfy $f (x) \geq c$ for some constant $c > 0$ , then the differential entropy becomes a smooth functional and consequently, the general technique for estimating smooth nonparametric functionals [51, 50, 43] can be directly applied here to achieve the minimax rates $n^{- \frac{4 s}{4 s + d}} + n^{- 1 / 2}$ . The main technical tools that enabled us to remove the conditions on how close the density could be from zero are the Besicovitch covering lemma (Lemma. 4) and the generalized Hardy–Littlewood maximal inequality.
We show that, for any fixed $k$ , the $k$ -nearest neighbor KL entropy estimator nearly achieves the minimax rates without knowing the smoothness parameter $s$ . In the functional estimation literature, designing estimators that can be theoretically proved to adapt to unknown levels of smoothness is usually achieved using the Lepski method [39, 22, 45, 44, 27], which is not known to be performing well in general in practice. On the other hand, a simple plug-in approach can achieves the rate of $n^{- s / (s + d)}$ , but only when $s$ is known [26]. The KL estimator is well known to exhibit excellent empirical performance, but existing theory has not yet demonstrated its near-“optimality” when the smoothness parameter $s$ is not known. Recent works [3, 52, 11] analyzed the performance of the KL estimator under various assumptions on how close the density could be to zero, with no matching lower bound up to logarithmic factors in general. Our work makes a step towards closing this gap and provides a theoretical explanation for the wide usage of the KL estimator in practice.

The rest of the paper is organized as follows. Section 2 is dedicated to the proof of Theorem 1. We discuss some future directions in Section 3.

1.2 Notations

For positive sequences $a_{γ}, b_{γ}$ , we use the notation $a_{γ} ≲_{α} b_{γ}$ to denote that there exists a universal constant $C$ that only depends on $α$ such that ${sup}_{γ} \frac{a_{γ}}{b_{γ}} \leq C$ , and $a_{γ} ≳_{α} b_{γ}$ is equivalent to $b_{γ} ≲_{α} a_{γ}$ . Notation $a_{γ} ≍_{α} b_{γ}$ is equivalent to $a_{γ} ≲_{α} b_{γ}$ and $b_{γ} ≲_{α} a_{γ}$ . We write $a_{γ} ≲ b_{γ}$ if the constant is universal and does not depend on any parameters. Notation $a_{γ} ≫ b_{γ}$ means that ${liminf}_{γ} \frac{a_{γ}}{b_{γ}} = \infty$ , and $a_{γ} ≪ b_{γ}$ is equivalent to $b_{γ} ≫ a_{γ}$ . We write $a \land b = min {a, b}$ and $a \lor b = max {a, b}$ .

2 Proof of Theorem 1

In this section, we will prove that

{(E {({^h}_{n, k} (X) - h (f))}_{n, k}^{2})}^{\frac{1}{2}} ≲_{s, L, d, k} n^{- \frac{s}{s + d}} ln (n + 1) + n^{- \frac{1}{} 2},

(8)

for any $f \in H_{d}^{s} (L; [0, 1]^{d})$ and $s \in (0, 2]$ . The proof consists two parts: (i) the upper bound of the bias in the form of $O_{s, L, d, k} (n^{- s / (s + d)} ln (n + 1))$

; (ii) the upper bound of the variance is

O_{s, L, d, k} (n^{- 1})

. Below we show the bias proof and relegate the variance proof to Appendix B.

First, we introduce the following notation

f_{t} (x)

=

\frac{μ (B (x, t))}{λ (B (x, t))} = \frac{1}{V_{d} t^{d}} \int_{u : | u - x | \leq t} f (u) d u .

(9)

Here $μ$

is the probability measure specified by density function

f

on the torus,

λ

is the Lebesgue measure on

R^{d}

, and

V_{d} = π^{d / 2} / Γ (1 + d / 2)

is the Lebesgue measure of the unit ball in

d

-dimensional Euclidean space. Hence

f_{t} (x)

is the average density of a neighborhood near

x

. We first state two main lemmas about

f_{t} (x)

which will be used later in the proof.

Lemma 1

If $f \in H_{d}^{s} (L; [0, 1]^{d})$ for some $0 < s \leq 2$ , then for any $x \in [0, 1]^{d}$ and $t > 0$ , we have

| f_{t} (x) - f (x) |

\leq

\frac{d L t^{s}}{s + d},

(10)

Lemma 2

If $f \in H_{d}^{s} (L; [0, 1]^{d})$ for some $0 < s \leq 2$ and $f (x) \geq 0$ for all $x \in [0, 1]^{d}$ , then for any $x$ and any $t > 0$ , we have

f (x)

≲_{s, L, d}

max {f_{t} (x), {(f_{t} (x) V_{d} t^{d})}_{t}^{s / (s + d)}},

(11)

Furthermore, $f (x) ≲_{s, L, d} 1$ .

We relegate the proof of Lemma 1 and Lemma 2 to Appendix C. Now we investigate the bias of ${^h}_{n, k} (X)$ . The following argument reduces the bias analysis of ${^h}_{n, k} (X)$ to a function analytic problem. For notation simplicity, we introduce a new random variable $X \sim f$ independent of ${X_{1}, \dots, X_{n}}$ and study $^hn+1,k({X1,…,Xn,X})$ . For every $x \in R^{d}$ , denote $R_{k} (x)$ by the $k$ -nearest neighbor distance from $x$ to ${X_{1}, X_{2}, \dots, X_{n}}$ under distance $d (x, y) = {min}_{m \in Z^{d}} ∥ m + x - y ∥$ , i.e., the $k$ -nearest neighbor distance on the torus. Then,

	$E[^hn+1,k({X1,…,Xn,X})]−h(f)$	(12)
$=$	$- ψ (k) + E [ln ((n + 1) λ (B (X, R_{k} (X))))] + E [ln f (X)]$	(13)
$=$	$E [ln (\frac{f (X) λ (B (X, R_{k} (X)))}{μ (B (X, R_{k} (X)))})] + E [ln ((n + 1) μ (B (X, R_{k} (X))))] - ψ (k)$	(14)
$=$	$E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}] + (E [ln ((n + 1) μ (B (X, R_{k} (X))))] - ψ (k)) .$	(15)

We first show that the second term $E [ln ((n + 1) μ (B (X, R_{k} (X))))] - ψ (k)$ can be universally controlled regardless of the smoothness of $f$ . Indeed, the random variable $μ (B (X, R_{k} (X))) \sim B e t a (k, n + 1 - k)$ [4, Chap. 1.2] and it was shown in [4, Theorem 7.2] that there exists a universal constant $C > 0$ such that

∣ ∣ E [ln ((n + 1) μ (B (X, R_{k} (X))))] - ψ (k) ∣ ∣

\leq

\frac{C}{n} .

(16)

Hence, it suffices to show that for $0 < s \leq 2$ ,

∣ ∣ ∣ ∣ E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}] ∣ ∣ ∣ ∣

≲_{s, L, d, k}

n^{- \frac{s}{s + d}} ln (n + 1) .

(17)

We split our analysis into two parts. Section 2.1 shows that $E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}] ≲_{s, L, d, k} n^{- \frac{s}{s + d}}$ and Section 2.2 shows that $E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}] ≲_{s, L, d, k} n^{- \frac{s}{s + d}} ln (n + 1)$ , which completes the proof.

2.1 Upper bound on $E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}]$

By the fact that $ln y \leq y - 1$ for any $y > 0$ , we have

	$E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}]$	$\leq$	$E [\frac{f_{R_{k} (X)} (X) - f (X)}{f (X)}]$		(18)
		$=$	$\int_{[0, 1]^{d} \cap {x : f (x) \neq 0}} (E [f_{R_{k} (x)} (x)] - f (x)) d x .$		(19)

Here the expectation is taken with respect to the randomness in $R_{k} (x) = {min}_{1 \leq i \leq n, m \in Z^{d}} ∥ m + X_{i} - x ∥, x \in R^{d}$ . Define function $g (x; f, n)$ as

g (x; f, n)

=

sup {u \geq 0 : V_{d} u^{d} f_{u} (x) \leq \frac{1}{n}},

(20)

$g (x; f, n)$ intuitively means the distance $R$ such that the probability mass $μ (B (x, R))$ within $R$ is $1 / n$ . Then for any $x \in [0, 1]^{d}$ , we can split $E [f_{R_{k} (x)} (x)] - f (x)$ into three terms as

$E [f_{R_{k} (x)} (x)] - f (x)$	$=$	$E [(f_{R_{k} (x)} (x) - f (x)) 1 (R_{k} (x) \leq n^{- 1 / (s + d)})]$	(21)
	$+$	$E [(f_{R_{k} (x)} (x) - f (x)) 1 (n^{- 1 / (s + d)} < R_{k} (x) \leq g (x; f, n))]$	(22)
	$+$	$E [(f_{R_{k} (x)} (x) - f (x)) 1 (R_{k} (x) > g (x; f, n) \lor n^{- 1 / (s + d)})]$	(23)
	$=$	$C_{1} + C_{2} + C_{3} .$	(24)

Now we handle three terms separately. Our goal is to show that for every $x \in [0, 1]$ , $C_{i} ≲_{s, L, d} n^{- s / (s + d)}$ for $i \in {1, 2, 3}$ . Then, taking the integral with respect to $x$ leads to the desired bound.

Term $C_{1}$ : whenever $R_{k} (x) \leq n^{- 1 / (s + d)}$ , by Lemma 1, we have

$| f_{R_{k} (x)} (x) - f (x) | \leq \frac{d L R_{k} (x)^{s}}{s + d} ≲_{s, L, d} n^{- s / (s + d)},$ (25)

which implies that

$C_{1} \leq E [∣ ∣ f_{R_{k} (x)} (x) - f (x) ∣ ∣ 1 (R_{k} (x) \leq n^{- 1 / (s + d)})] ≲_{s, L, d} n^{- s / (s + d)} .$ (26)
Term $C_{2}$ : whenever $R_{k} (x)$ satisfies that $n^{- 1 / (s + d)} < R_{k} (x) \leq g (x; f, n)$ , by definition of $g (x; f, n)$ , we have $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) \leq \frac{1}{n}$ , which implies that

$f_{R_{k} (x)} (x) \leq \frac{1}{n V_{d} R_{k} (x)^{d}} \leq \frac{1}{n V_{d} n^{- d / (s + d)}} ≲_{s, L, d} n^{- s / (s + d)} .$ (27)

It follows from Lemma 2 that in this case

$f (x)$ $≲_{s, L, d}$ $f_{R_{k} (x)} (x) \lor {(f_{R_{k} (x)} (x) V_{d} R_{k} (x)^{d})}_{R_{k} (x)}^{s / (s + d)}$ (28)

$\leq$ $n^{- s / (s + d)} \lor n^{- s / (s + d)} = n^{- s / (s + d)} .$ (29)

Hence, we have

$C_{2}$ $=$ $E [(f_{R_{k} (x)} (x) - f (x)) 1 (n^{- 1 / (s + d)} < R_{k} (x) \leq g (x; f, n))]$ (30)

$\leq$ $E [(f_{R_{k} (x)} (x) + f (x)) 1 (n^{- 1 / (s + d)} < R_{k} (x) \leq g (x; f, n))]$ (31)

$≲_{s, L, d}$ $n^{- s / (s + d)} .$ (32)
Term $C_{3}$ : we have

$C_{3}$ $\leq$ $E [(f_{R_{k} (x)} (x) + f (x)) 1 (R_{k} (x) > g (x; f, n) \lor n^{- 1 / (s + d)})] .$ (33)

For any $x$ such that $R_{k} (x) > n^{- 1 / (s + d)}$ , we have

$f_{R_{k} (x)} (x)$ $≲_{s, L, d}$ $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) n^{d / (s + d)},$ (34)

and by Lemma 2,

$f (x)$ $≲_{s, L, d}$ $f_{R_{k} (x)} (x) \lor (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)}$ (35)

$\leq$ $f_{R_{k} (x)} (x) + (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)} .$ (36)

Hence,

$f (x) + f_{R_{k} (x)} (x)$ $≲_{s, L, d}$ $2 f_{R_{k} (x)} (x) + (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)}$ (38)

$≲_{s, L, d}$ $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) n^{d / (s + d)} + (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)}$

$≲_{s, L, d}$ $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) n^{d / (s + d)},$ (39)

where in the last step we have used the fact that $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) > n^{- 1}$ since $R_{k} (x) > g (x; f, n)$ . Finally, we have

$C_{3}$ $≲_{s, L, d}$ (40)

$=$ $n^{d / (s + d)} E [(V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x)) 1 (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) > 1 / n)] .$ (41)

Note that $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) \sim B e t a (k, n + 1 - k)$ , and if $Y \sim B e t a (k, n + 1 - k)$ , we have

$E [Y^{2}] = {(\frac{k}{n + 1})}^{2} + \frac{k (n + 1 - k)}{(n + 1)^{2} (n + 2)} ≲_{k} \frac{1}{n^{2}} .$ (42)

Notice that $E [Y 1 (Y > 1 / n)] \leq n E [Y^{2}]$ . Hence, we have

$C_{3}$ $≲_{s, L, d}$ $n^{d / (s + d)} n E [(V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{2}]$ (43)

$≲_{s, L, d, k}$ $\frac{n^{d / (s + d)} n}{n^{2}} = n^{- s / (s + d)} .$ (44)

2.2 Upper bound on $E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}]$

By splitting the term into two parts, we have

$E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}]$	$=$	$E [\int_{[0, 1]^{d} \cap {x : f (x) \neq 0}} f (x) ln \frac{f (x)}{f_{R_{k} (x)} (x)} d x]$	(45)
	$=$	$E [\int_{A} f (x) ln \frac{f (x)}{f_{R_{k} (x)} (x)} 1 (f_{R_{k} (x)} (x) > n^{- s / (s + d)}) d x]$	(46)
	$+$	$E [\int_{A} f (x) ln \frac{f (x)}{f_{R_{k} (x)} (x)} 1 (f_{R_{k} (x)} (x) \leq n^{- s / (s + d)}) d x]$	(47)
	$=$	$C_{4} + C_{5} .$	(48)

here we denote $A = [0, 1]^{d} \cap {x : f (x) \neq 0}$ for simplicity of notation. For the term $C_{4}$ , we have

$C_{4}$	$\leq$	$E [\int_{A} f (x) (\frac{f (x) - f_{R_{k} (x)} (x)}{f_{R_{k} (x)} (x)}) 1 (f_{R_{k} (x)} (x) > n^{- s / (s + d)}) d x]$	(49)
	$=$	$E [\int_{A} \frac{(f (x) - f_{R_{k} (x)} (x))^{2}}{f_{R_{k} (x)} (x)} 1 (f_{R_{k} (x)} (x) > n^{- s / (s + d)}) d x]$	(50)
	$+$		(51)
	$\leq$	$n^{s / (s + d)} E [\int_{A} {(f (x) - f_{R_{k} (x)} (x))}_{R_{k} (x)}^{2} d x] + E [\int_{A} (f (x) - f_{R_{k} (x)} (x)) d x] .$	(52)

In the proof of upper bound of $E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}]$ , we have shown that $E [f_{R_{k} (x)} (x) - f (x)] ≲_{s, L, d, k} n^{- s / (s + d)}$ for any $x \in A$ . Similarly as in the proof of upper bound of $E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}]$ , we have $E [(f_{R_{k} (x)} (x) - f (x))^{2}] ≲_{s, L, d, k} n^{- 2 s / (s + d)}$ for every $x \in A$ . Therefore, we have

C_{4}

≲_{s, L, d, k}

n^{s / (s + d)} n^{- 2 s / (s + d)} + n^{- s / (s + d)} ≲_{s, L, d, k} n^{- s / (s + d)} .

(53)

Now we consider $C_{5}$ . We conjecture that $C_{5} ≲_{s, L, d, k} n^{- s / (s + d)}$ in this case, but we were not able to prove it. Below we prove that $C_{5} ≲_{s, L, d, k} n^{- s / (s + d)} ln (n + 1)$ . Define the function

M (x)

=

sup t > 0 \frac{1}{f_{t} (x)} .

(54)

Since $f_{R_{k} (x)} (x) \leq n^{- s / (s + d)}$ , we have $M (x) = {sup}_{t > 0} (1 / f_{t} (x)) \geq 1 / f_{R_{k} (x)} (x) \geq n^{s / (s + d)}$ . Denote ${ln}^{+} (y) = max {ln (y), 0}$ for any $y > 0$ , therefore, we have that

$C_{5}$	$\leq$	$E [\int_{A} f (x) {ln}^{+} (\frac{f (x)}{f_{R_{k} (x)} (x)}) 1 (f_{R_{k} (x)} (x) \leq n^{- s / (s + d)}) d x]$	(55)
	$\leq$	$E [\int_{A} f (x) {ln}^{+} (\frac{f (x)}{f_{R_{k} (x)} (x)}) 1 (M (x) \geq n^{s / (s + d)}) d x]$	(56)
	$\leq$	$\int_{A} f (x) E [{ln}^{+} (\frac{1}{(n + 1) V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x)})] 1 (M (x) \geq n^{s / (s + d)}) d x$	(57)
	$+$	$\int_{A} f (x) E [{ln}^{+} ((n + 1) V_{d} R_{k} (x)^{d} f (x))] 1 (M (x) \geq n^{s / (s + d)}) d x$	(58)
	$=$	$C_{51} + C_{52},$	(59)

where the last inequality uses the fact ${ln}^{+} (x y) \leq {ln}^{+} x + {ln}^{+} y$ for all $x, y > 0$ . As for $C_{51}$ , since $V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) \sim B e t a (k, n + 1 - k)$ , and for $Y \sim B e t a (k, n + 1 - k)$ , we have

$E [{ln}^{+} (\frac{1}{(n + 1) Y})]$	$=$	$\int_{0}^{\frac{1}{n + 1}} ln (\frac{1}{(n + 1) x}) p_{Y} (x) d x$	(60)
	$=$	$E [ln (\frac{1}{(n + 1) Y})] + \int_{\frac{1}{n + 1}}^{1} ln ((n + 1) x) p_{Y} (x) d x$	(61)
	$\leq$	$E [ln (\frac{1}{(n + 1) Y})] + ln (n + 1) \int_{\frac{1}{n + 1}}^{1} p_{Y} (x) d x$	(62)
	$\leq$	$E [ln (\frac{1}{(n + 1) Y})] + ln (n + 1)$	(63)
	$\leq$	$ln (n + 1)$	(64)

where in the last inequality we used the fact that $E [ln (\frac{1}{(n + 1) Y})] = ψ (n + 1) - ψ (k) - ln (n + 1) \leq 0$ for any $k \geq 1$ . Hence,

C_{51}

≲_{s, L, d}

ln (n + 1) \int_{A} f (x) 1 (M (x) \geq n^{s / (s + d)}) d x .

(65)

Now we introduce the following lemma, which is proved in Appendix C.

Lemma 3

Let $μ_{1}, μ_{2}$ be two Borel measures that are finite on the bounded Borel sets of $R^{d}$ . Then, for all $t > 0$ and any Borel set $A \subset R^{d}$ ,

μ_{1} ({x \in A : sup 0 < ρ \leq D (\frac{μ_{2} (B (x, ρ))}{μ_{1} (B (x, ρ))}) > t})

\leq

\frac{C_{d}}{t} μ_{2} (A_{D}) .

(66)

Here $C_{d} > 0$ is a constant that depends only on the dimension $d$ and

A_{D} = {x : \exists y \in A, | y - x | \leq D} .

(67)

Applying the second part of Lemma 3 with $μ_{2}$ being the Lebesgue measure and $μ_{1}$ being the measure specified by $f (x)$ on the torus, we can view the function $M (x)$ as

M (x) = sup 0 < ρ \leq 1 / 2 \frac{μ_{2} (B (x, ρ))}{μ_{1} (B (x, ρ))} .

(68)

Taking $A = [0, 1]^{d} \cap {x : f (x) \neq 0}, t = n^{s / (s + d)}$ , then $μ_{2} (A_{\frac{1}{2}}) \leq 2^{d}$ , so we know that

The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal

Authors

Minimax Optimal Estimation of KL Divergence for Continuous Distributions

One-Nearest-Neighbor Search is All You Need for Minimax Optimal Regression and Classification

Demystifying Fixed k-Nearest Neighbor Information Estimators

Minimax Rate Optimal Adaptive Nearest Neighbor Classification and Regression

On Testing for Parameters in Ising Models

Minimax Analysis for Inverse Risk in Nonparametric Planer Invertible Regression

On the consistency of the Kozachenko-Leonenko entropy estimate

1 Introduction

1.1 Main Contribution

Theorem 1

Theorem 2

Remark 1

1.2 Notations

2 Proof of Theorem 1

Lemma 1

Lemma 2

2.1 Upper bound on $E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}]$

2.2 Upper bound on $E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}]$

Lemma 3

	$f (x)$	$≲_{s, L, d}$	$f_{R_{k} (x)} (x) \lor {(f_{R_{k} (x)} (x) V_{d} R_{k} (x)^{d})}_{R_{k} (x)}^{s / (s + d)}$		(28)
		$\leq$	$n^{- s / (s + d)} \lor n^{- s / (s + d)} = n^{- s / (s + d)} .$		(29)

$C_{2}$	$=$	$E [(f_{R_{k} (x)} (x) - f (x)) 1 (n^{- 1 / (s + d)} < R_{k} (x) \leq g (x; f, n))]$	(30)
	$\leq$	$E [(f_{R_{k} (x)} (x) + f (x)) 1 (n^{- 1 / (s + d)} < R_{k} (x) \leq g (x; f, n))]$	(31)
	$≲_{s, L, d}$	$n^{- s / (s + d)} .$	(32)

	$f (x)$	$≲_{s, L, d}$	$f_{R_{k} (x)} (x) \lor (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)}$		(35)
		$\leq$	$f_{R_{k} (x)} (x) + (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)} .$		(36)

$f (x) + f_{R_{k} (x)} (x)$	$≲_{s, L, d}$	$2 f_{R_{k} (x)} (x) + (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)}$	(38)
	$≲_{s, L, d}$	$V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) n^{d / (s + d)} + (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{s / (s + d)}$

	$≲_{s, L, d}$	$V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) n^{d / (s + d)},$	(39)

	$C_{3}$	$≲_{s, L, d}$			(40)
		$=$	$n^{d / (s + d)} E [(V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x)) 1 (V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x) > 1 / n)] .$		(41)

	$C_{3}$	$≲_{s, L, d}$	$n^{d / (s + d)} n E [(V_{d} R_{k} (x)^{d} f_{R_{k} (x)} (x))^{2}]$		(43)
		$≲_{s, L, d, k}$	$\frac{n^{d / (s + d)} n}{n^{2}} = n^{- s / (s + d)} .$		(44)

The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal

Authors

Related Research

Minimax Optimal Estimation of KL Divergence for Continuous Distributions

One-Nearest-Neighbor Search is All You Need for Minimax Optimal Regression and Classification

Demystifying Fixed k-Nearest Neighbor Information Estimators

Minimax Rate Optimal Adaptive Nearest Neighbor Classification and Regression

On Testing for Parameters in Ising Models

Minimax Analysis for Inverse Risk in Nonparametric Planer Invertible Regression

On the consistency of the Kozachenko-Leonenko entropy estimate

This week in AI

1 Introduction

1.1 Main Contribution

Theorem 1

Theorem 2

Remark 1

1.2 Notations

2 Proof of Theorem 1

Lemma 1

Lemma 2

2.1 Upper bound on E[lnfRk(X)(X)f(X)]

2.2 Upper bound on E[lnf(X)fRk(X)(X)]

Lemma 3

2.1 Upper bound on $E [ln \frac{f_{R_{k} (X)} (X)}{f (X)}]$

2.2 Upper bound on $E [ln \frac{f (X)}{f_{R_{k} (X)} (X)}]$