Confidence Modeling for Neural Semantic Parsing

by   Li Dong, et al.

In this work we focus on confidence modeling for neural semantic parsers which are built upon sequence-to-sequence models. We outline three major causes of uncertainty, and design various metrics to quantify these factors. These metrics are then used to estimate confidence scores that indicate whether model predictions are likely to be correct. Beyond confidence estimation, we identify which parts of the input contribute to uncertain predictions allowing users to interpret their model, and verify or refine its input. Experimental results show that our confidence model significantly outperforms a widely used method that relies on posterior probability, and improves the quality of interpretation compared to simply relying on attention scores.


Failure Prediction by Confidence Estimation of Uncertainty-Aware Dirichlet Networks

Reliably assessing model confidence in deep learning and predicting erro...

Joint Dermatological Lesion Classification and Confidence Modeling with Uncertainty Estimation

Deep learning has played a major role in the interpretation of dermoscop...

Modeling Confidence in Sequence-to-Sequence Models

Recently, significant improvements have been achieved in various natural...

Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model

As neural network classifiers are deployed in real-world applications, i...

Grammatical Sequence Prediction for Real-Time Neural Semantic Parsing

While sequence-to-sequence (seq2seq) models achieve state-of-the-art per...

On Modeling ASR Word Confidence

We present a new method for computing ASR word confidences that effectiv...

Usable Region Estimate for Assessing Practical Usability of Medical Image Segmentation Models

We aim to quantitatively measure the practical usability of medical imag...