DeepAI AI Chat
Log In Sign Up

Automatic Open-World Reliability Assessment

11/11/2020
by   Mohsen Jafarzadeh, et al.
0

Image classification in the open-world must handle out-of-distribution (OOD) images. Systems should ideally reject OOD images, or they will map atop of known classes and reduce reliability. Using open-set classifiers that can reject OOD inputs can help. However, optimal accuracy of open-set classifiers depend on the frequency of OOD data. Thus, for either standard or open-set classifiers, it is important to be able to determine when the world changes and increasing OOD inputs will result in reduced system reliability. However, during operations, we cannot directly assess accuracy as there are no labels. Thus, the reliability assessment of these classifiers must be done by human operators, made more complex because networks are not 100 failures are to be expected. To automate this process, herein, we formalize the open-world recognition reliability problem and propose multiple automatic reliability assessment policies to address this new problem using only the distribution of reported scores/probability data. The distributional algorithms can be applied to both classic classifiers with SoftMax as well as the open-world Extreme Value Machine (EVM) to provide automated reliability assessment. We show that all of the new algorithms significantly outperform detection using the mean of SoftMax.

READ FULL TEXT
07/08/2020

A Critical Evaluation of Open-World Machine Learning

Open-world machine learning (ML) combines closed-world models trained on...
08/29/2018

Extreme Value Theory for Open Set Classification - GPD and GEV Classifiers

Classification tasks usually assume that all possible classes are presen...
07/27/2021

Energy-Based Open-World Uncertainty Modeling for Confidence Calibration

Confidence calibration is of great importance to the reliability of deci...
10/09/2022

Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples

Modern image classifiers achieve high predictive accuracy, but the predi...
05/20/2019

Testing Deep Neural Network based Image Classifiers

Image classification is an important task in today's world with many app...
05/30/2022

Exploring the Open World Using Incremental Extreme Value Machines

Dynamic environments require adaptive applications. One particular machi...