DeepAI AI Chat
Log In Sign Up

Post-Selection Confidence Bounds for Prediction Performance

10/24/2022
by   Pascal Rink, et al.
Universität Bremen
0

In machine learning, the selection of a promising model from a potentially large number of competing models and the assessment of its generalization performance are critical tasks that need careful consideration. Typically, model selection and evaluation are strictly separated endeavors, splitting the sample at hand into a training, validation, and evaluation set, and only compute a single confidence interval for the prediction performance of the final selected model. We however propose an algorithm how to compute valid lower confidence bounds for multiple models that have been selected based on their prediction performances in the evaluation set by interpreting the selection problem as a simultaneous inference problem. We use bootstrap tilting and a maxT-type multiplicity correction. The approach is universally applicable for any combination of prediction models, any model selection strategy, and any prediction performance measure that accepts weights. We conducted various simulation experiments which show that our proposed approach yields lower confidence bounds that are at least comparably good as bounds from standard approaches, and that reliably reach the nominal coverage probability. In addition, especially when sample size is small, our proposed approach yields better performing prediction models than the default selection of only one model for evaluation does.

READ FULL TEXT

page 3

page 4

page 5

page 7

page 10

page 11

page 12

page 13

11/06/2017

Two sources of poor coverage of confidence intervals after model selection

We compare the following two sources of poor coverage of post-model-sele...
01/09/2018

Test Error Estimation after Model Selection Using Validation Error

When performing supervised learning with the model selected using valida...
07/15/2020

Selective Inference for Additive and Linear Mixed Models

This work addresses the problem of conducting valid inference for additi...
04/12/2022

Hold-out estimates of prediction models for Markov processes

We consider the selection of prediction models for Markovian time series...
09/12/2018

Prediction out-of-sample using block shrinkage estimators: model selection and predictive inference

In a linear regression model with random design, we consider a family of...
11/08/2019

A multiple testing framework for diagnostic accuracy studies with co-primary endpoints

Major advances have been made regarding the utilization of artificial in...