DeepAI AI Chat
Log In Sign Up

Meta Pattern Concern Score: A Novel Metric for Customizable Evaluation of Multi-classification

by   Yanyun Wang, et al.

Classifiers have been widely implemented in practice, while how to evaluate them properly remains a problem. Commonly used two types of metrics respectively based on confusion matrix and loss function have different advantages in flexibility and mathematical completeness, while they struggle in different dilemmas like the insensitivity to slight improvements or the lack of customizability in different tasks. In this paper, we propose a novel metric named Meta Pattern Concern Score based on the abstract representation of the probabilistic prediction, as well as the targeted design for processing negative classes in multi-classification and reducing the discreteness of metric value, to achieve advantages of both the two kinds of metrics and avoid their weaknesses. Our metric provides customizability to pick out the model for specific requirements in different practices, and make sure it is also fine under traditional metrics at the same time. Evaluation in four kinds of models and six datasets demonstrates the effectiveness and efficiency of our metric, and a case study shows it can select a model to reduce 0.53 misclassifications by sacrificing only 0.04


page 1

page 2

page 3

page 4


Extending F_1 metric, probabilistic approach

This article explores the extension of well-known F_1 score used for ass...

sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification

Multiclass multilabel classification refers to the task of attributing m...

Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment

In most machine learning training paradigms a fixed, often handcrafted, ...

Ontology-aware Learning and Evaluation for Audio Tagging

This study defines a new evaluation metric for audio tagging tasks to ov...

A Heaviside Function Approximation for Neural Network Binary Classification

Neural network binary classifiers are often evaluated on metrics like ac...

MetricOpt: Learning to Optimize Black-Box Evaluation Metrics

We study the problem of directly optimizing arbitrary non-differentiable...

Does the evaluation stand up to evaluation? A first-principle approach to the evaluation of classifiers

How can one meaningfully make a measurement, if the meter does not confo...

Code Repositories



view repo