Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach

by   Emmanouil A. Platanios, et al.

We propose an efficient method to estimate the accuracy of classifiers using only unlabeled data. We consider a setting with multiple classification problems where the target classes may be tied together through logical constraints. For example, a set of classes may be mutually exclusive, meaning that a data instance can belong to at most one of them. The proposed method is based on the intuition that: (i) when classifiers agree, they are more likely to be correct, and (ii) when the classifiers make a prediction that violates the constraints, at least one classifier must be making an error. Experiments on four real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs. The results emphasize the utility of logical constraints in estimating accuracy, thus validating our intuition.



There are no comments yet.


page 1

page 2

page 3

page 4


Alternate Estimation of a Classifier and the Class-Prior from Positive and Unlabeled Data

We consider a problem of learning a binary classifier only from positive...

Optimally Combining Classifiers Using Unlabeled Data

We develop a worst-case analysis of aggregation of classifier ensembles ...

Object Recognition Based on Amounts of Unlabeled Data

This paper proposes a novel semi-supervised method on object recognition...

Performance-Agnostic Fusion of Probabilistic Classifier Outputs

We propose a method for combining probabilistic outputs of classifiers t...

Learning Classifiers on Positive and Unlabeled Data with Policy Gradient

Existing algorithms aiming to learn a binary classifier from positive (P...

A Prediction Tournament Paradox

In a prediction tournament, contestants "forecast" by asserting a numeri...

Assessing binary classifiers using only positive and unlabeled data

Assessing the performance of a learned model is a crucial part of machin...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.