DeepAI AI Chat
Log In Sign Up

Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances

by   Takyoung Kim, et al.
sejong university
Korea University

The primary purpose of dialogue state tracking (DST), a critical component of an end-to-end conversational system, is to build a model that responds well to real-world situations. Although we often change our minds during ordinary conversations, current benchmark datasets do not adequately reflect such occurrences and instead consist of over-simplified conversations, in which no one changes their mind during a conversation. As the main question inspiring the present study,“Are current benchmark datasets sufficiently diverse to handle casual conversations in which one changes their mind?” We found that the answer is “No” because simply injecting template-based turnback utterances significantly degrades the DST model performance. The test joint goal accuracy on the MultiWOZ decreased by over 5%p when the simplest form of turnback utterance was injected. Moreover, the performance degeneration worsens when facing more complicated turnback situations. However, we also observed that the performance rebounds when a turnback is appropriately included in the training dataset, implying that the problem is not with the DST models but rather with the construction of the benchmark dataset.


MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines

MultiWOZ is a recently-released multidomain dialogue dataset spanning 7 ...

Dialogue-based neural learning to estimate the sentiment of a next upcoming utterance

In a conversation, humans use changes in a dialogue to predict safety-cr...

CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Dialogue state trackers have made significant progress on benchmark data...

CDConv: A Benchmark for Contradiction Detection in Chinese Conversations

Dialogue contradiction is a critical issue in open-domain dialogue syste...

Dial2Desc: End-to-end Dialogue Description Generation

We first propose a new task named Dialogue Description (Dial2Desc). Unli...

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions

Existing conversational systems are mostly agent-centric, which assumes ...

Generating Strategic Dialogue for Negotiation with Theory of Mind

We propose a framework to integrate the concept of Theory of Mind (ToM) ...