Extrapolation Towards Imaginary 0-Nearest Neighbour and Its Improved Convergence Rate
share
k-nearest neighbour (k-NN) is one of the simplest and most widely-used methods for supervised classification, that predicts a query's label by taking weighted ratio of observed labels of k objects nearest to the query. The weights and the parameter k ∈N regulate its bias-variance trade-off, and the trade-off implicitly affects the convergence rate of the excess risk for the k-NN classifier; several existing studies considered selecting optimal k and weights to obtain faster convergence rate. Whereas k-NN with non-negative weights has been developed widely, it was proved that negative weights are essential for eradicating the bias terms and attaining optimal convergence rate. However, computation of the optimal weights requires solving entangled equations. Thus, other simpler approaches that can find optimal real-valued weights are appreciated in practice. In this paper, we propose multiscale k-NN (MS-k-NN), that extrapolates unweighted k-NN estimators from several k > 1 values to k=0, thus giving an imaginary 0-NN estimator. MS-k-NN implicitly corresponds to an adaptive method for finding favorable real-valued weights, and we theoretically prove that the MS-k-NN attains the improved rate, that coincides with the existing optimal rate under some conditions.
READ FULL TEXT