Calibrated Surrogate Maximization of Linear-fractional Utility in Binary Classification

05/29/2019
by   Han Bao, et al.
0

Complex classification performance metrics such as the F_β-measure and Jaccard index are often used, in order to handle class-imbalanced cases such as information retrieval and image segmentation. These performance metrics are not decomposable, that is, they cannot be expressed in a per-example manner, which hinders a straightforward application of the M-estimation widely used in supervised learning. In this paper, we consider linear-fractional metrics, which are a family of classification performance metrics that encompasses many standard metrics such as the F_β-measure and Jaccard index, and propose methods to directly maximize performances under those metrics. A clue to tackle their direct optimization is a calibrated surrogate utility, which is a tractable lower bound of the true utility function representing a given metric. We characterize necessary conditions which make the surrogate maximization coincide with the maximization of the true utility. To the best of our knowledge, this is the first surrogate calibration analysis for the linear-fractional metrics. We also propose gradient-based optimization algorithms and show their practical usefulness in experiments.

READ FULL TEXT
research
04/03/2021

A surrogate loss function for optimization of F_β score in binary classification with imbalanced data

The F_β score is a commonly used measure of classification performance, ...
research
01/04/2023

Network Utility Maximization with Unknown Utility Functions: A Distributed, Data-Driven Bilevel Optimization Approach

Fair resource allocation is one of the most important topics in communic...
research
05/07/2015

Optimal Decision-Theoretic Classification Using Non-Decomposable Performance Metrics

We provide a general theoretical analysis of expected out-of-sample util...
research
05/28/2020

Calibrated Surrogate Losses for Adversarially Robust Classification

Adversarially robust classification seeks a classifier that is insensiti...
research
08/19/2022

Classification Performance Metric Elicitation and its Applications

Given a learning problem with real-world tradeoffs, which cost function ...
research
06/02/2021

Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

A/B tests have been widely adopted across industries as the golden rule ...
research
07/18/2022

Consistent Polyhedral Surrogates for Top-k Classification and Variants

Top-k classification is a generalization of multiclass classification us...

Please sign up or login with your details

Forgot password? Click here to reset