Bad practices in evaluation methodology relevant to class-imbalanced problems

12/04/2018
by   Jan Brabec, et al.
0

For research to go in the right direction, it is essential to be able to compare and quantify performance of different algorithms focused on the same problem. Choosing a suitable evaluation metric requires deep understanding of the pursued task along with all of its characteristics. We argue that in the case of applied machine learning, proper evaluation metric is the basic building block that should be in the spotlight and put under thorough examination. Here, we address tasks with class imbalance, in which the class of interest is the one with much lower number of samples. We encountered non-insignificant amount of recent papers, in which improper evaluation methods are used, borrowed mainly from the field of balanced problems. Such bad practices may heavily bias the results in favour of inappropriate algorithms and give false expectations of the state of the field.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/15/2020

On Model Evaluation under Non-constant Class Imbalance

Many real-world classification problems are significantly class-imbalanc...
research
09/04/2019

Metric Learning from Imbalanced Data

A key element of any machine learning algorithm is the use of a function...
research
12/17/2022

Balanced Split: A new train-test data splitting strategy for imbalanced datasets

Classification data sets with skewed class proportions are called imbala...
research
10/15/2017

A systematic study of the class imbalance problem in convolutional neural networks

In this study, we systematically investigate the impact of class imbalan...
research
05/01/2019

Class Imbalance Techniques for High Energy Physics

A common problem in high energy physics is extracting a signal from a mu...
research
04/11/2018

The Evolution of User-Selected Passwords: A Quantitative Analysis of Publicly Available Datasets

The aim of this work is to study the evolution of password selection amo...
research
06/08/2021

Realistic Evaluation Principles for Cross-document Coreference Resolution

We point out that common evaluation practices for cross-document corefer...

Please sign up or login with your details

Forgot password? Click here to reset