DeepAI AI Chat
Log In Sign Up

Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances

08/28/2021
by   Takyoung Kim, et al.
sejong university
Korea University
0

The primary purpose of dialogue state tracking (DST), a critical component of an end-to-end conversational system, is to build a model that responds well to real-world situations. Although we often change our minds during ordinary conversations, current benchmark datasets do not adequately reflect such occurrences and instead consist of over-simplified conversations, in which no one changes their mind during a conversation. As the main question inspiring the present study,“Are current benchmark datasets sufficiently diverse to handle casual conversations in which one changes their mind?” We found that the answer is “No” because simply injecting template-based turnback utterances significantly degrades the DST model performance. The test joint goal accuracy on the MultiWOZ decreased by over 5%p when the simplest form of turnback utterance was injected. Moreover, the performance degeneration worsens when facing more complicated turnback situations. However, we also observed that the performance rebounds when a turnback is appropriately included in the training dataset, implying that the problem is not with the DST models but rather with the construction of the benchmark dataset.

READ FULL TEXT
07/02/2019

MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines

MultiWOZ is a recently-released multidomain dialogue dataset spanning 7 ...
04/21/2021

Dialogue-based neural learning to estimate the sentiment of a next upcoming utterance

In a conversation, humans use changes in a dialogue to predict safety-cr...
10/24/2020

CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Dialogue state trackers have made significant progress on benchmark data...
10/16/2022

CDConv: A Benchmark for Contradiction Detection in Chinese Conversations

Dialogue contradiction is a critical issue in open-domain dialogue syste...
11/01/2018

Dial2Desc: End-to-end Dialogue Description Generation

We first propose a new task named Dialogue Description (Dial2Desc). Unli...
10/24/2020

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions

Existing conversational systems are mostly agent-centric, which assumes ...
10/20/2020

Generating Strategic Dialogue for Negotiation with Theory of Mind

We propose a framework to integrate the concept of Theory of Mind (ToM) ...