Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

10/06/2020
by   Carel van Niekerk, et al.
0

The ability to accurately track what happens during a conversation is essential for the performance of a dialogue system. Current state-of-the-art multi-domain dialogue state trackers achieve just over 55 current go-to benchmark, which means that in almost every second dialogue turn they place full confidence in an incorrect dialogue state. Belief trackers, on the other hand, maintain a distribution over possible dialogue states. However, they lack in performance compared to dialogue state trackers, and do not produce well calibrated distributions. In this work we present state-of-the-art performance in calibration for multi-domain dialogue belief trackers using a calibrated ensemble of models. Our resulting dialogue belief tracker also outperforms previous dialogue belief tracking models in terms of accuracy.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/20/2019

Teacher-Student Framework Enhanced Multi-domain Dialogue Generation

Dialogue systems dealing with multi-domain tasks are highly required. Ho...
10/24/2020

CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Dialogue state trackers have made significant progress on benchmark data...
12/30/2020

Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness

Open-domain dialogue agents have vastly improved, but still confidently ...
09/09/2021

Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance

The ability to identify and resolve uncertainty is crucial for the robus...
10/22/2019

Scalable Neural Dialogue State Tracking

A Dialogue State Tracker (DST) is a key component in a dialogue system a...
07/20/2021

Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System

A medical dialogue system is essential for healthcare service as providi...
05/29/2018

Fully Statistical Neural Belief Tracking

This paper proposes an improvement to the existing data-driven Neural Be...