Optimal Sampling Density for Nonparametric Regression

by   Danny Panknin, et al.

We propose a novel active learning strategy for regression, which is model-agnostic, robust against model mismatch, and interpretable. Assuming that a small number of initial samples are available, we derive the optimal training density that minimizes the generalization error of local polynomial smoothing (LPS) with its kernel bandwidth tuned locally: We adopt the mean integrated squared error (MISE) as a generalization criterion, and use the asymptotic behavior of the MISE as well as thelocally optimal bandwidths (LOB) – the bandwidth function that minimizes MISE in the asymptotic limit. The asymptotic expression of our objective then reveals the dependence of the MISE on the training density, enabling analytic minimization. As a result, we obtain the optimal training density in a closed-form. The almost model-free nature of our approach should encode raw properties of the target problem, and thus provide a robust and model-agnostic active learning strategy. Furthermore, the obtained training density factorizes the influence of local function complexity, noise leveland test density in a transparent and interpretable way. We validate our theory in numerical simulations, and show that the proposed active learning method outperforms the existing state-of-the-art model-agnostic approaches.



page 24


Bayesian bandwidth estimation for local linear fitting in nonparametric regression models

This paper presents a Bayesian sampling approach to bandwidth estimation...

Data-Based Optimal Bandwidth for Kernel Density Estimation of Statistical Samples

It is a common practice to evaluate probability density function or matt...

Axiomatic Approach to Variable Kernel Density Estimation

Variable kernel density estimation allows the approximation of a probabi...

Robust and Active Learning for Deep Neural Network Regression

We describe a gradient-based method to discover local error maximizers o...

Nonparametric modal regression

Modal regression estimates the local modes of the distribution of Y give...

Floodgate: inference for model-free variable importance

Many modern applications seek to understand the relationship between an ...

Target-Independent Active Learning via Distribution-Splitting

To reduce the label complexity in Agnostic Active Learning (A^2 algorith...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.