Distance-based Confidence Score for Neural Network Classifiers

09/28/2017
by   Amit Mandelbaum, et al.
0

The reliable measurement of confidence in classifiers' predictions is very important for many applications and is, therefore, an important part of classifier design. Yet, although deep learning has received tremendous attention in recent years, not much progress has been made in quantifying the prediction confidence of neural network classifiers. Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with prohibitive computational costs. In this paper we propose a simple, scalable method to achieve a reliable confidence score, based on the data embedding derived from the penultimate layer of the network. We investigate two ways to achieve desirable embeddings, by using either a distance-based loss or Adversarial Training. We then test the benefits of our method when used for classification error prediction, weighting an ensemble of classifiers, and novelty detection. In all tasks we show significant improvement over traditional, commonly used confidence scores.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/05/2020

Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model

As neural network classifiers are deployed in real-world applications, i...
research
04/21/2016

Novelty Detection in MultiClass Scenarios with Incomplete Set of Class Labels

We address the problem of novelty detection in multiclass scenarios wher...
research
03/20/2019

Performance Measurement for Deep Bayesian Neural Network

Deep Bayesian neural network has aroused a great attention in recent yea...
research
05/28/2018

Confidence Prediction for Lexicon-Free OCR

Having a reliable accuracy score is crucial for real world applications ...
research
05/19/2019

Leveraging Semantic Embeddings for Safety-Critical Applications

Semantic Embeddings are a popular way to represent knowledge in the fiel...
research
12/13/2021

WOOD: Wasserstein-based Out-of-Distribution Detection

The training and test data for deep-neural-network-based classifiers are...
research
10/14/2022

Confidence estimation of classification based on the distribution of the neural network output layer

One of the most common problems preventing the application of prediction...

Please sign up or login with your details

Forgot password? Click here to reset