Estimation of prediction error with known covariate shift

05/04/2022
by   Hui Xu, et al.
0

In supervised learning, the estimation of prediction error on unlabeled test data is an important task. Existing methods are usually built on the assumption that the training and test data are sampled from the same distribution, which is often violated in practice. As a result, traditional estimators like cross-validation (CV) will be biased and this may result in poor model selection. In this paper, we assume that we have a test dataset in which the feature values are available but not the outcome labels, and focus on a particular form of distributional shift called "covariate shift". We propose an alternative method based on parametric bootstrap of the target of conditional error. Empirically, our method outperforms CV for both simulation and real data example across different modeling tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/19/2023

Information Geometrically Generalized Covariate Shift Adaptation

Many machine learning methods assume that the training and test data fol...
research
11/02/2019

Model Specification Test with Unlabeled Data: Approach from Covariate Shift

We propose a novel framework of the model specification test in regressi...
research
02/24/2022

An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift

A common assumption in semi-supervised learning is that the labeled, unl...
research
08/27/2019

Locally Optimized Random Forests

Standard supervised learning procedures are validated against a test set...
research
07/01/2021

Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection

While many real-world data streams imply that they change frequently in ...
research
03/10/2018

Webly Supervised Learning with Category-level Semantic Information

As tons of photos are being uploaded to public websites (e.g., Flickr, B...
research
10/14/2020

A Distribution-Free Test of Covariate Shift Using Conformal Prediction

Covariate shift is a common and important assumption in transfer learnin...

Please sign up or login with your details

Forgot password? Click here to reset