Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals

06/01/2021
by   Jiri Navratil, et al.
0

Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to a specific operating point, making evaluation and comparison across different studies difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a simple reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper describes the corresponding algorithm, provides a theoretical analysis, and demonstrates its utility in multiple scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/11/2022

The Implicit Delta Method

Epistemic uncertainty quantification is a crucial part of drawing credib...
research
05/11/2022

Demystifying Inferential Models: A Fiducial Perspective

Inferential models have recently gained in popularity for valid uncertai...
research
07/16/2021

Uncertainty Prediction for Machine Learning Models of Material Properties

Uncertainty quantification in Artificial Intelligence (AI)-based predict...
research
06/18/2022

Nonparametric Multi-shape Modeling with Uncertainty Quantification

The modeling and uncertainty quantification of closed curves is an impor...
research
06/25/2019

Simultaneous Prediction Intervals for Patient-Specific Survival Curves

Accurate models of patient survival probabilities provide important info...
research
12/15/2020

Learning Prediction Intervals for Model Performance

Understanding model performance on unlabeled data is a fundamental chall...
research
05/28/2023

Speech Intelligibility Assessment of Dysarthric Speech by using Goodness of Pronunciation with Uncertainty Quantification

This paper proposes an improved Goodness of Pronunciation (GoP) that uti...

Please sign up or login with your details

Forgot password? Click here to reset