How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?

12/21/2022
by   Kazuma Hashimoto, et al.
0

Text-to-text generation models have increasingly become the go-to solution for a wide variety of sequence labeling tasks (e.g., entity extraction and dialog slot filling). While most research has focused on the labeling accuracy, a key aspect – of vital practical importance – has slipped through the cracks: understanding model confidence. More specifically, we lack a principled understanding of how to reliably gauge the confidence of a model in its predictions for each labeled span. This paper aims to provide some empirical insights on estimating model confidence for generative sequence labeling. Most notably, we find that simply using the decoder's output probabilities is not the best in realizing well-calibrated confidence estimates. As verified over six public datasets of different tasks, we show that our proposed approach – which leverages statistics from top-k predictions by a beam search – significantly reduces calibration errors of the predictions of a generative sequence labeling model.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/26/2022

Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents

This paper introduces a new information extraction model for business do...
research
10/09/2020

iobes: A Library for Span-Level Processing

Many tasks in natural language processing, such as named entity recognit...
research
09/29/2022

GROOT: Corrective Reward Optimization for Generative Sequential Labeling

Sequential labeling is a fundamental NLP task, forming the backbone of m...
research
01/15/2017

Neural Models for Sequence Chunking

Many natural language understanding (NLU) tasks, such as shallow parsing...
research
06/09/2016

Sequence-to-Sequence Learning as Beam-Search Optimization

Sequence-to-Sequence (seq2seq) modeling has rapidly become an important ...
research
09/15/2020

Augmented Natural Language for Generative Sequence Labeling

We propose a generative framework for joint sequence labeling and senten...
research
09/29/2021

Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration

An increasingly common use case for machine learning models is augmentin...

Please sign up or login with your details

Forgot password? Click here to reset