Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction

10/06/2022
by   Sangwoo Park, et al.
0

Conventional frequentist learning is known to yield poorly calibrated models that fail to reliably quantify the uncertainty of their decisions. Bayesian learning can improve calibration, but formal guarantees apply only under restrictive assumptions about correct model specification. Conformal prediction (CP) offers a general framework for the design of set predictors with calibration guarantees that hold regardless of the underlying data generation mechanism. However, when training data are limited, CP tends to produce large, and hence uninformative, predicted sets. This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. Unlike prior work, the proposed meta-learning scheme, referred to as meta-XB, (i) builds on cross-validation-based CP, rather than the less efficient validation-based CP; and (ii) preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees. Finally, meta-XB is extended to adaptive non-conformal scores, which are shown empirically to further enhance marginal per-input calibration.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/10/2022

Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction

AI tools can be useful to address model deficits in the design of commun...
research
12/15/2022

Calibrating AI Models for Wireless Communications via Conformal Prediction

When used in complex engineered systems, such as communication networks,...
research
08/02/2021

Learning to Learn to Demodulate with Uncertainty Quantification via Bayesian Meta-Learning

Meta-learning, or learning to learn, offers a principled framework for f...
research
02/17/2021

Few-shot Conformal Prediction with Auxiliary Tasks

We develop a novel approach to conformal prediction when the target task...
research
06/17/2021

Meta-Calibration: Meta-Learning of Model Calibration Using Differentiable Expected Calibration Error

Calibration of neural networks is a topical problem that is becoming inc...
research
02/27/2023

Causal isotonic calibration for heterogeneous treatment effects

We propose causal isotonic calibration, a novel nonparametric method for...
research
06/05/2023

On training locally adaptive CP

We address the problem of making Conformal Prediction (CP) intervals loc...

Please sign up or login with your details

Forgot password? Click here to reset