Binary classification models with "Uncertain" predictions

by   Damjan Krstajic, et al.

Binary classification models which can assign probabilities to categories such as "the tissue is 75 likely to be toxic" are well understood statistically, but their utility as an input to decision making is less well explored. We argue that users need to know which is the most probable outcome, how likely that is to be true and, in addition, whether the model is capable enough to provide an answer. It is the last case, where the potential outcomes of the model explicitly include "don't know" that is addressed in this paper. Including this outcome would better separate those predictions that can lead directly to a decision from those where more data is needed. Where models produce an "Uncertain" answer similar to a human reply of "don't know" or "50:50" in the examples we refer to earlier, this would translate to actions such as "operate on tumour" or "remove compound from use" where the models give a "more true than not" answer. Where the models judge the result "Uncertain" the practical decision might be "carry out more detailed laboratory testing of compound" or "commission new tissue analyses". The paper presents several examples where we first analyse the effect of its introduction, then present a methodology for separating "Uncertain" from binary predictions and finally, we provide arguments for its use in practice.


page 1

page 2

page 3

page 4


Natural Language Generation enhances human decision-making with uncertain information

Decision-making is often dependent on uncertain data, e.g. data associat...

Linking a predictive model to causal effect estimation

A predictive model makes outcome predictions based on some given feature...

Bias Mitigation Methods for Binary Classification Decision-Making Systems: Survey and Recommendations

Bias mitigation methods for binary classification decision-making system...

Decision-Making under Miscalibration

ML-based predictions are used to inform consequential decisions about in...

Reliable Decision Support using Counterfactual Models

Making a good decision involves considering the likely outcomes under ea...

A Tale Of Two Long Tails

As machine learning models are increasingly employed to assist human dec...

A critical assessment of conformal prediction methods applied in binary classification settings

In recent years there has been an increase in the number of scientific p...

Please sign up or login with your details

Forgot password? Click here to reset