Meta Pattern Concern Score: A Novel Metric for Customizable Evaluation of Multi-classification

09/14/2022
by   Yanyun Wang, et al.
0

Classifiers have been widely implemented in practice, while how to evaluate them properly remains a problem. Commonly used two types of metrics respectively based on confusion matrix and loss function have different advantages in flexibility and mathematical completeness, while they struggle in different dilemmas like the insensitivity to slight improvements or the lack of customizability in different tasks. In this paper, we propose a novel metric named Meta Pattern Concern Score based on the abstract representation of the probabilistic prediction, as well as the targeted design for processing negative classes in multi-classification and reducing the discreteness of metric value, to achieve advantages of both the two kinds of metrics and avoid their weaknesses. Our metric provides customizability to pick out the model for specific requirements in different practices, and make sure it is also fine under traditional metrics at the same time. Evaluation in four kinds of models and six datasets demonstrates the effectiveness and efficiency of our metric, and a case study shows it can select a model to reduce 0.53 misclassifications by sacrificing only 0.04

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/23/2023

Ties Matter: Modifying Kendall's Tau for Modern Metric Meta-Evaluation

Kendall's tau is frequently used to meta-evaluate how well machine trans...
research
10/21/2022

Extending F_1 metric, probabilistic approach

This article explores the extension of well-known F_1 score used for ass...
research
08/24/2021

sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification

Multiclass multilabel classification refers to the task of attributing m...
research
05/15/2019

Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment

In most machine learning training paradigms a fixed, often handcrafted, ...
research
11/22/2022

Ontology-aware Learning and Evaluation for Audio Tagging

This study defines a new evaluation metric for audio tagging tasks to ov...
research
06/17/2018

Learning to Evaluate Image Captioning

Evaluation metrics for image captioning face two challenges. Firstly, co...
research
07/09/2018

Detecting Levels of Depression in Text Based on Metrics

Depression is one of the most common and a major concern for society. Pr...

Please sign up or login with your details

Forgot password? Click here to reset