What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts

10/31/2021
by   James D. Finch, et al.
0

Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/04/2019

Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems

User engagement is a critical metric for evaluating the quality of open-...
research
04/04/2019

Improving Dialogue State Tracking by Discerning the Relevant Context

A typical conversation comprises of multiple turns between participants ...
research
11/18/2019

Multi-domain Conversation Quality Evaluation via User Satisfaction Estimation

An automated metric to evaluate dialogue quality is vital for optimizing...
research
06/01/2019

Multi-Turn Beam Search for Neural Dialogue Modeling

In neural dialogue modeling, a neural network is trained to predict the ...
research
04/26/2022

Understanding User Satisfaction with Task-oriented Dialogue Systems

Dialogue systems are evaluated depending on their type and purpose. Two ...
research
12/24/2020

I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling

To quantify how well natural language understanding models can capture c...
research
10/24/2020

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions

Existing conversational systems are mostly agent-centric, which assumes ...

Please sign up or login with your details

Forgot password? Click here to reset