PAC Prediction Sets for Meta-Learning

07/06/2022
by   Sangdon Park, et al.
0

Uncertainty quantification is a key component of machine learning models targeted at safety-critical systems such as in healthcare or autonomous vehicles. We study this problem in the context of meta learning, where the goal is to quickly adapt a predictor to new tasks. In particular, we propose a novel algorithm to construct PAC prediction sets, which capture uncertainty via sets of labels, that can be adapted to new tasks with only a few training examples. These prediction sets satisfy an extension of the typical PAC guarantee to the meta learning setting; in particular, the PAC guarantee holds with high probability over future tasks. We demonstrate the efficacy of our approach on four datasets across three application domains: mini-ImageNet and CIFAR10-C in the visual domain, FewRel in the language domain, and the CDC Heart Dataset in the medical domain. In particular, our prediction sets satisfy the PAC guarantee while having smaller size compared to other baselines that also satisfy this guarantee.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/18/2023

PAC Neural Prediction Set Learning to Quantify the Uncertainty of Generative Language Models

Uncertainty learning and quantification of models are crucial tasks to e...
research
05/28/2021

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

Despite recent advances in its theoretical understanding, there still re...
research
02/12/2021

PAC-BUS: Meta-Learning Bounds via PAC-Bayes and Uniform Stability

We are motivated by the problem of providing strong generalization guara...
research
02/13/2020

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

Meta-learning can successfully acquire useful inductive biases from data...
research
04/15/2022

Towards PAC Multi-Object Detection and Tracking

Accurately detecting and tracking multi-objects is important for safety-...
research
12/31/2019

PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction

We propose an algorithm combining calibrated prediction and generalizati...
research
11/12/2020

Ensuring Actionable Recourse via Adversarial Training

As machine learning models are increasingly deployed in high-stakes doma...

Please sign up or login with your details

Forgot password? Click here to reset