Statistical Comparisons of Classifiers by Generalized Stochastic Dominance

09/05/2022
by   Christoph Jansen, et al.
0

Although being a question in the very methodological core of machine learning, there is still no unanimous consensus on how to compare classifiers. Every comparison framework is confronted with (at least) three fundamental challenges: the multiplicity of quality criteria, the multiplicity of data sets and the randomness/arbitrariness of the selection of data sets. In this paper, we add a fresh view to the vivid debate by adopting recent developments in decision theory. Our resulting framework, based on so-called preference systems, ranks classifiers by a generalized concept of stochastic dominance, which powerfully circumvents the cumbersome, and often even self-contradictory, reliance on aggregates. Moreover, we show that generalized stochastic dominance can be operationalized by solving easy-to-handle linear programs and statistically tested by means of an adapted two-sample observation-randomization test. This indeed yields a powerful framework for the statistical comparison of classifiers with respect to multiple quality criteria simultaneously. We illustrate and investigate our framework in a simulation study and with standard benchmark data sets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/07/2017

External Evaluation of Event Extraction Classifiers for Automatic Pathway Curation: An extended study of the mTOR pathway

This paper evaluates the impact of various event extraction systems on a...
research
08/15/2023

How to Simulate Realistic Survival Data? A Simulation Study to Compare Realistic Simulation Models

In statistics, it is important to have realistic data sets available for...
research
09/16/2021

Probability-driven scoring functions in combining linear classifiers

Although linear classifiers are one of the oldest methods in machine lea...
research
01/04/2023

A Comparison of Fundamental Methods for Iso-surface Extraction

In this paper four fundamental methods for an iso-surface extraction are...
research
05/10/2017

Analysing Data-To-Text Generation Benchmarks

Recently, several data-sets associating data to text have been created t...
research
04/19/2023

Depth Functions for Partial Orders with a Descriptive Analysis of Machine Learning Algorithms

We propose a framework for descriptively analyzing sets of partial order...
research
08/30/2019

Partitioned integrators for thermodynamic parameterization of neural networks

Stochastic Gradient Langevin Dynamics, the "unadjusted Langevin algorith...

Please sign up or login with your details

Forgot password? Click here to reset