Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance

09/09/2021
by   Carel van Niekerk, et al.
0

The ability to identify and resolve uncertainty is crucial for the robustness of a dialogue system. Indeed, this has been confirmed empirically on systems that utilise Bayesian approaches to dialogue belief tracking. However, such systems consider only confidence estimates and have difficulty scaling to more complex settings. Neural dialogue systems, on the other hand, rarely take uncertainties into account. They are therefore overconfident in their decisions and less robust. Moreover, the performance of the tracking task is often evaluated in isolation, without consideration of its effect on the downstream policy optimisation. We propose the use of different uncertainty measures in neural belief tracking. The effects of these measures on the downstream task of policy optimisation are evaluated by adding selected measures of uncertainty to the feature space of the policy and training policies through interaction with a user simulator. Both human and simulated user results show that incorporating these measures leads to improvements both of the performance and of the robustness of the downstream dialogue policy. This highlights the importance of developing neural dialogue belief trackers that take uncertainty into account.

READ FULL TEXT
research
11/30/2017

Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation

In statistical dialogue management, the dialogue manager learns a policy...
research
07/29/2022

"Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking

While communicating with a user, a task-oriented dialogue system has to ...
research
10/06/2020

Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

The ability to accurately track what happens during a conversation is es...
research
05/17/2018

Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems

User Simulators are one of the major tools that enable offline training ...
research
06/12/2016

Neural Belief Tracker: Data-Driven Dialogue State Tracking

One of the core components of modern spoken dialogue systems is the beli...
research
05/29/2018

Fully Statistical Neural Belief Tracking

This paper proposes an improvement to the existing data-driven Neural Be...
research
11/04/2020

Hybrid Supervised Reinforced Model for Dialogue Systems

This paper presents a recurrent hybrid model and training procedure for ...

Please sign up or login with your details

Forgot password? Click here to reset