What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

06/10/2021
by   Weijian Deng, et al.
0

Understanding classifier decision under novel environments is central to the community, and a common practice is evaluating it on labeled test sets. However, in real-world testing, image annotations are difficult and expensive to obtain, especially when the test environment is changing. A natural question then arises: given a trained classifier, can we evaluate its accuracy on varying unlabeled test sets? In this work, we train semantic classification and rotation prediction in a multi-task way. On a series of datasets, we report an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson's Correlation r > 0.88). This finding allows us to utilize linear regression to estimate classifier performance from the accuracy of rotation prediction which can be obtained on the test set through the freely generated rotation labels.

READ FULL TEXT

page 4

page 6

research
09/07/2022

Improving Self-supervised Learning for Out-of-distribution Task via Auxiliary Classifier

In real world scenarios, out-of-distribution (OOD) datasets may have a l...
research
07/06/2020

Are Labels Necessary for Classifier Accuracy Evaluation?

To calculate the model accuracy on a computer vision task, e.g., object ...
research
07/09/2016

Classifier Risk Estimation under Limited Labeling Resources

In this paper we propose strategies for estimating performance of a clas...
research
01/10/2017

Efficient Image Set Classification using Linear Regression based Image Reconstruction

We propose a novel image set classification technique using linear regre...
research
09/18/2018

Is rotation forest the best classifier for problems with continuous features?

Rotation forest is a tree based ensemble that performs transforms on sub...
research
03/26/2018

Image Set Classification for Low Resolution Surveillance

This paper proposes a novel image set classification technique based on ...
research
02/06/2016

Classification Accuracy as a Proxy for Two Sample Testing

When data analysts train a classifier and check if its accuracy is signi...

Please sign up or login with your details

Forgot password? Click here to reset