Predictive Value Generalization Bounds

07/09/2020
by   Keshav Vemuri, et al.
0

In this paper, we study a bi-criterion framework for assessing scoring functions in the context of binary classification. The positive and negative predictive values (ppv and npv, respectively) are conditional probabilities of the true label matching a classifier's predicted label. The usual classification error rate is a linear combination of these probabilities, and therefore, concentration inequalities for the error rate do not yield confidence intervals for the two separate predictive values. We study generalization properties of scoring functions with respect to predictive values by deriving new distribution-free large deviation and uniform convergence bounds. The latter bound is stated in terms of a measure of function class complexity that we call the order coefficient; we relate this combinatorial quantity to the VC-subgraph dimension.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/25/2013

Supersparse Linear Integer Models for Predictive Scoring Systems

We introduce Supersparse Linear Integer Models (SLIM) as a tool to creat...
research
06/18/2020

Distribution-free binary classification: prediction sets, confidence intervals and calibration

We study three notions of uncertainty quantification—calibration, confid...
research
10/19/2020

Variance-adaptive confidence sequences by betting

This paper derives confidence intervals (CI) and time-uniform confidence...
research
08/03/2023

Distribution-Free Inference for the Regression Function of Binary Classification

One of the key objects of binary classification is the regression functi...
research
08/16/2019

Algorithms and Complexity for Functions on General Domains

Error bounds and complexity bounds in numerical analysis and information...
research
07/19/2023

Repeated Observations for Classification

We study the problem nonparametric classification with repeated observat...
research
07/26/2018

Rademacher Generalization Bounds for Classifier Chains

In this paper, we propose a new framework to study the generalization pr...

Please sign up or login with your details

Forgot password? Click here to reset