Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking

03/07/2022
by   Takyoung Kim, et al.
0

Dialogue state tracking (DST) aims to extract essential information from multi-turn dialogue situations and take appropriate actions. A belief state, one of the core pieces of information, refers to the subject and its specific content, and appears in the form of . The trained model predicts "accumulated" belief states in every turn, and joint goal accuracy and slot accuracy are mainly used to evaluate the prediction; however, we specify that the current evaluation metrics have a critical limitation when evaluating belief states accumulated as the dialogue proceeds, especially in the most used MultiWOZ dataset. Additionally, we propose relative slot accuracy to complement existing metrics. Relative slot accuracy does not depend on the number of predefined slots, and allows intuitive evaluation by assigning relative scores according to the turn of each dialogue. This study also encourages not solely the reporting of joint goal accuracy, but also various complementary metrics in DST tasks for the sake of a realistic evaluation.

READ FULL TEXT

page 4

page 13

research
06/16/2021

Coreference Augmentation for Multi-Domain Task-Oriented Dialogue State Tracking

Dialogue State Tracking (DST), which is the process of inferring user go...
research
04/07/2022

Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances

Dialogue State Tracking (DST) is primarily evaluated using Joint Goal Ac...
research
12/03/2018

Toward Scalable Neural Dialogue State Tracking Model

The latency in the current neural based dialogue state tracking models p...
research
10/24/2020

CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Dialogue state trackers have made significant progress on benchmark data...
research
09/16/2022

SF-DST: Few-Shot Self-Feeding Reading Comprehension Dialogue State Tracking with Auxiliary Task

Few-shot dialogue state tracking (DST) model tracks user requests in dia...
research
09/22/2020

CREDIT: Coarse-to-Fine Sequence Generation for Dialogue State Tracking

In dialogue systems, a dialogue state tracker aims to accurately find a ...
research
11/10/2019

Efficient Dialogue State Tracking by Selectively Overwriting Memory

Recent works in dialogue state tracking (DST) focus on an open vocabular...

Please sign up or login with your details

Forgot password? Click here to reset