Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

by   Carel van Niekerk, et al.

The ability to accurately track what happens during a conversation is essential for the performance of a dialogue system. Current state-of-the-art multi-domain dialogue state trackers achieve just over 55 current go-to benchmark, which means that in almost every second dialogue turn they place full confidence in an incorrect dialogue state. Belief trackers, on the other hand, maintain a distribution over possible dialogue states. However, they lack in performance compared to dialogue state trackers, and do not produce well calibrated distributions. In this work we present state-of-the-art performance in calibration for multi-domain dialogue belief trackers using a calibrated ensemble of models. Our resulting dialogue belief tracker also outperforms previous dialogue belief tracking models in terms of accuracy.


page 1

page 2

page 3

page 4


Teacher-Student Framework Enhanced Multi-domain Dialogue Generation

Dialogue systems dealing with multi-domain tasks are highly required. Ho...

CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Dialogue state trackers have made significant progress on benchmark data...

Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness

Open-domain dialogue agents have vastly improved, but still confidently ...

Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance

The ability to identify and resolve uncertainty is crucial for the robus...

Scalable Neural Dialogue State Tracking

A Dialogue State Tracker (DST) is a key component in a dialogue system a...

Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System

A medical dialogue system is essential for healthcare service as providi...

Fully Statistical Neural Belief Tracking

This paper proposes an improvement to the existing data-driven Neural Be...