Predicting Classification Accuracy when Adding New Unobserved Classes

by   Yuli Slavutsky, et al.

Multiclass classifiers are often designed and evaluated only on a sample from the classes on which they will eventually be applied. Hence, their final accuracy remains unknown. In this work we study how a classifier's performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes. For this, we define a measure of separation between correct and incorrect classes that is independent of the number of classes: the reversed ROC (rROC), which is obtained by replacing the roles of classes and data-points in the common ROC. We show that the classification accuracy is a function of the rROC in multiclass classifiers, for which the learned representation of data from the initial class sample remains unchanged when new classes are added. Using these results we formulate a robust neural-network-based algorithm, CleaneX, which learns to estimate the accuracy of such classifiers on arbitrarily large sets of classes. Our method achieves remarkably better predictions than current state-of-the-art methods on both simulations and real datasets of object detection, face recognition, and brain decoding.



There are no comments yet.


page 12


Classification-Reconstruction Learning for Open-Set Recognition

Open-set classification is a problem of handling `unknown' classes that ...

How many faces can be recognized? Performance extrapolation for multi-class classification

The difficulty of multi-class classification generally increases with th...

Data Separability for Neural Network Classifiers and the Development of a Separability Index

In machine learning, the performance of a classifier depends on both the...

Beyond cross-entropy: learning highly separable feature distributions for robust and accurate classification

Deep learning has shown outstanding performance in several applications ...

Neural eliminators and classifiers

Classification may not be reliable for several reasons: noise in the dat...

Incremental Deep Neural Network Learning using Classification Confidence Thresholding

Most modern neural networks for classification fail to take into account...

LOBO -- Evaluation of Generalization Deficiencies in Twitter Bot Classifiers

Botnets in online social networks are increasingly often affecting the r...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.