Evaluation Measures for Quantification: An Axiomatic Approach

09/06/2018
by   Fabrizio Sebastiani, et al.
0

Quantification is the task of estimating, given a set σ of unlabelled items and a set of classes C={c_1, ..., c_|C|}, the prevalence (or `relative frequency') in σ of each class c_i∈C. While quantification may in principle be solved by classifying each item in σ and counting how many such items have been labelled with c_i, it has long been shown that this `classify and count' (CC) method yields suboptimal quantification accuracy. As a result, quantification is no longer considered a mere byproduct of classification, and has evolved as a task of its own. While the scientific community has devoted a lot of attention to devising more accurate quantification methods, it has not devoted much to discussing what properties an evaluation measure for quantification (EMQ) should enjoy, and which EMQs should be adopted as a result. This paper lies down a number of interesting properties that an EMQ may or may not enjoy, discusses if (and when) each of these properties is desirable, surveys the EMQs that have been used so far, and discusses whether they enjoy or not the above properties. As a result of this investigation, some of the EMQs that have been used in the literature turn out to be severely unfit, while others emerge as closer to what the quantification community actually needs. However, a significant result is that no existing EMQ satisfies all the properties identified as desirable, thus indicating that more research is needed in order to identify (or synthesize) a truly adequate EMQ.

READ FULL TEXT

page 13

page 14

research
09/04/2018

A Recurrent Neural Network for Sentiment Quantification

Quantification is a supervised learning task that consists in predicting...
research
02/28/2016

Does quantification without adjustments work?

Classification is the task of predicting the class labels of objects bas...
research
12/25/2021

Stance Quantification: Definition of the Problem

Stance detection is commonly defined as the automatic process of determi...
research
04/16/2019

Cross-Lingual Sentiment Quantification

We discuss Cross-Lingual Text Quantification (CLTQ), the task of perform...
research
03/04/2021

A Comparative Evaluation of Quantification Methods

Quantification represents the problem of predicting class distributions ...
research
05/13/2016

Online Optimization Methods for the Quantification Problem

The estimation of class prevalence, i.e., the fraction of a population t...
research
12/14/2017

Rate of Change Analysis for Interestingness Measures

The use of Association Rule Mining techniques in diverse contexts and do...

Please sign up or login with your details

Forgot password? Click here to reset