How Nonconformity Functions and Difficulty of Datasets Impact the Efficiency of Conformal Classifiers

08/12/2021
by   Marharyta Aleksandrova, et al.
0

The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the systems can output multiple class labels instead of one. It is also known from the literature, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the inverse probability (or hinge loss) allows minimizing the average number of predicted labels, and margin results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relationship holds or not. Additionally, we propose a successful method to combine the properties of these two nonconformity functions. The experimental evaluation is done using 11 real and 5 synthetic datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/14/2020

Training conformal predictors

Efficiency criteria for conformal prediction, such as observed fuzziness...
research
09/02/2016

Least Ambiguous Set-Valued Classifiers with Bounded Error Levels

In most classification tasks there are observations that are ambiguous a...
research
08/02/2021

Multiple Classifiers Based Maximum Classifier Discrepancy for Unsupervised Domain Adaptation

Adversarial training based on the maximum classifier discrepancy between...
research
09/21/2021

Signal Classification using Smooth Coefficients of Multiple wavelets

Classification of time series signals has become an important construct ...
research
05/24/2021

Learning Security Classifiers with Verified Global Robustness Properties

Recent works have proposed methods to train classifiers with local robus...
research
05/18/2018

A case study of hurdle and generalized additive models in astronomy: the escape of ionizing radiation

The dark ages of the Universe end with the formation of the first genera...
research
01/29/2019

Trading-off Accuracy and Energy of Deep Inference on Embedded Systems: A Co-Design Approach

Deep neural networks have seen tremendous success for different modaliti...

Please sign up or login with your details

Forgot password? Click here to reset