Distribution-free Prediction Sets Adaptive to Unknown Covariate Shift
share
Predicting sets of outcomes – instead of unique outcomes – is a promising solution to uncertainty quantification in statistical learning. Despite a rich literature on constructing prediction sets with statistical guarantees, adapting to unknown covariate shift – a prevalent issue in practice – poses a serious challenge and has yet to be fully solved. In this paper, we propose a novel flexible distribution-free method, PredSet-1Step, to construct prediction sets that can efficiently adapt to unknown covariate shift. We formally show that our method is asymptotically probably approximately correct, having well-calibrated coverage error with high confidence for large samples. We illustrate that it achieves nominal coverage in a number of experiments and a data set concerning HIV risk prediction in a South African cohort study. Our theory hinges on a new bound for the convergence rate of the coverage of Wald confidence intervals based on general asymptotically linear estimators. This is a technical tool of independent interest.
READ FULL TEXT