Confidence Modeling for Neural Semantic Parsing

05/11/2018
by   Li Dong, et al.
0

In this work we focus on confidence modeling for neural semantic parsers which are built upon sequence-to-sequence models. We outline three major causes of uncertainty, and design various metrics to quantify these factors. These metrics are then used to estimate confidence scores that indicate whether model predictions are likely to be correct. Beyond confidence estimation, we identify which parts of the input contribute to uncertain predictions allowing users to interpret their model, and verify or refine its input. Experimental results show that our confidence model significantly outperforms a widely used method that relies on posterior probability, and improves the quality of interpretation compared to simply relying on attention scores.

READ FULL TEXT
10/19/2020

Failure Prediction by Confidence Estimation of Uncertainty-Aware Dirichlet Networks

Reliably assessing model confidence in deep learning and predicting erro...
07/19/2021

Joint Dermatological Lesion Classification and Confidence Modeling with Uncertainty Estimation

Deep learning has played a major role in the interpretation of dermoscop...
10/04/2019

Modeling Confidence in Sequence-to-Sequence Models

Recently, significant improvements have been achieved in various natural...
10/05/2020

Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model

As neural network classifiers are deployed in real-world applications, i...
07/25/2019

Grammatical Sequence Prediction for Real-Time Neural Semantic Parsing

While sequence-to-sequence (seq2seq) models achieve state-of-the-art per...
07/22/2019

On Modeling ASR Word Confidence

We present a new method for computing ASR word confidences that effectiv...
07/01/2022

Usable Region Estimate for Assessing Practical Usability of Medical Image Segmentation Models

We aim to quantitatively measure the practical usability of medical imag...