DeepAI
Log In Sign Up

A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues

03/01/2021
by   Ziming Li, et al.
0

The evaluation of multi-turn dialogues remains challenging. The common approach of labeling the user satisfaction with the experience on the dialogue level does not reflect the task's difficulty. Therefore assigning the same experience score to two tasks with different complexity levels is misleading. Another approach, which suggests evaluating each dialogue turn independently, ignores each turn's long-term influence over the final user experience with dialogue. We instead develop a new method to estimate the turn-level satisfaction for dialogue, which is context-sensitive and has a long-term view. Our approach is data-driven which makes it easily personalized. The interactions between users and dialogue systems are formulated using a budget consumption setup. We assume the user has an initial interaction budget for a conversation based on the task complexity, and each dialogue turn has a cost. When the task is completed or the budget has been run out, the user will quit the interaction. We demonstrate the effectiveness of our method by extensive experimentation with a simulated dialogue platform and a realistic dialogue dataset.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/26/2022

Understanding User Satisfaction with Task-oriented Dialogue Systems

Dialogue systems are evaluated depending on their type and purpose. Two ...
10/06/2020

Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations

Dialogue level quality estimation is vital for optimizing data driven di...
11/18/2019

Multi-domain Conversation Quality Evaluation via User Satisfaction Estimation

An automated metric to evaluate dialogue quality is vital for optimizing...
08/19/2019

Domain-Independent turn-level Dialogue Quality Evaluation via User Satisfaction Estimation

An automated metric to evaluate dialogue quality is vital for optimizing...
12/08/2021

FreeTalky: Don't Be Afraid! Conversations Made Easier by a Humanoid Robot using Persona-based Dialogue

We propose a deep learning-based foreign language learning platform, nam...
04/26/2020

Endowing Empathetic Dialogue Systems with Personas

Empathetic dialogue systems have been shown to improve user satisfaction...
01/13/2021

Is the User Enjoying the Conversation? A Case Study on the Impact on the Reward Function

The impact of user satisfaction in policy learning task-oriented dialogu...