DeepAI AI Chat
Log In Sign Up

Relaxed Conformal Prediction Cascades for Efficient Inference Over Many Labels

by   Adam Fisch, et al.

Providing a small set of promising candidates in place of a single prediction is well-suited for many open-ended classification tasks. Conformal Prediction (CP) is a technique for creating classifiers that produce a valid set of predictions that contains the true answer with arbitrarily high probability. In practice, however, standard CP can suffer from both low predictive and computational efficiency during inference—i.e., the predicted set is both unusably large, and costly to obtain. This is particularly pervasive in the considered setting, where the correct answer is not unique and the number of total possible answers is high. In this work, we develop two simple and complementary techniques for improving both types of efficiencies. First, we relax CP validity to arbitrary criterions of success—allowing our framework to make more efficient predictions while remaining "equivalently correct." Second, we amortize cost by conformalizing prediction cascades, in which we aggressively prune implausible labels early on by using progressively stronger classifiers—while still guaranteeing marginal coverage. We demonstrate the empirical effectiveness of our approach for multiple applications in natural language processing and computational chemistry for drug discovery.


page 1

page 2

page 3

page 4


Few-shot Conformal Prediction with Auxiliary Tasks

We develop a novel approach to conformal prediction when the target task...

RR-CP: Reliable-Region-Based Conformal Prediction for Trustworthy Medical Image Classification

Conformal prediction (CP) generates a set of predictions for a given tes...

Concepts and Applications of Conformal Prediction in Computational Drug Discovery

Estimating the reliability of individual predictions is key to increase ...

Nearest Neighbor Classifiers over Incomplete Information: From Certain Answers to Certain Predictions

Machine learning (ML) applications have been thriving recently, largely ...

Conformal Predictor for Improving Zero-shot Text Classification Efficiency

Pre-trained language models (PLMs) have been shown effective for zero-sh...

Learning Optimal Conformal Classifiers

Modern deep learning based classifiers show very high accuracy on test d...

Conformal Prediction Intervals for Remaining Useful Lifetime Estimation

The main objective of Prognostics and Health Management is to estimate t...