Minimax Rate Optimal Adaptive Nearest Neighbor Classification and Regression

by   Puning Zhao, et al.

k Nearest Neighbor (kNN) method is a simple and popular statistical method for classification and regression. For both classification and regression problems, existing works have shown that, if the distribution of the feature vector has bounded support and the probability density function is bounded away from zero in its support, the convergence rate of the standard kNN method, in which k is the same for all test samples, is minimax optimal. On the contrary, if the distribution has unbounded support, we show that there is a gap between the convergence rate achieved by the standard kNN method and the minimax bound. To close this gap, we propose an adaptive kNN method, in which different k is selected for different samples. Our selection rule does not require precise knowledge of the underlying distribution of features. The new proposed method significantly outperforms the standard one. We characterize the convergence rate of the proposed adaptive method, and show that it matches the minimax lower bound.



There are no comments yet.


page 1

page 2

page 3

page 4


Analysis of KNN Density Estimation

We analyze the ℓ_1 and ℓ_∞ convergence rates of k nearest neighbor densi...

One-Nearest-Neighbor Search is All You Need for Minimax Optimal Regression and Classification

Recently, Qiao, Duan, and Cheng (2019) proposed a distributed nearest-ne...

Stabilized Nearest Neighbor Classifier and Its Statistical Properties

The stability of statistical analysis is an important indicator for repr...

The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal

We analyze the Kozachenko--Leonenko (KL) nearest neighbor estimator for ...

Mathematical Properties of Continuous Ranked Probability Score Forecasting

The theoretical advances on the properties of scoring rules over the pas...

Enhanced Nearest Neighbor Classification for Crowdsourcing

In machine learning, crowdsourcing is an economical way to label a large...

A minimax near-optimal algorithm for adaptive rejection sampling

Rejection Sampling is a fundamental Monte-Carlo method. It is used to sa...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.