Helpfulness and Fairness of Task-Oriented Dialogue Systems

05/25/2022
by   Jiao Sun, et al.
0

Task-oriented dialogue systems aim to answer questions from users and provide immediate help. Therefore, how humans perceive their helpfulness is important. However, neither the human-perceived helpfulness of task-oriented dialogue systems nor its fairness implication has been studied yet. In this paper, we define a dialogue response as helpful if it is relevant coherent, useful, and informative to a query and study computational measurements of helpfulness. Then, we propose utilizing the helpfulness level of different groups to gauge the fairness of a dialogue system. To study this, we collect human annotations for the helpfulness of dialogue responses and build a classifier that can automatically determine the helpfulness of a response. We design experiments under 3 information-seeking scenarios and collect instances for each from Wikipedia. With collected instances, we use carefully-constructed questions to query the state-of-the-art dialogue systems. Through analysis, we find that dialogue systems tend to be more helpful for highly-developed countries than less-developed countries, uncovering a fairness issue underlying these dialogue systems.

READ FULL TEXT

page 7

page 8

research
10/14/2022

Q-TOD: A Query-driven Task-oriented Dialogue System

Existing pipelined task-oriented dialogue systems usually have difficult...
research
05/27/2020

Should Answer Immediately or Wait for Further Information? A Novel Wait-or-Answer Task and Its Predictive Approach

Different people have different habits of describing their intents in co...
research
05/23/2023

Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues

Task-oriented dialogue systems aim at providing users with task-specific...
research
10/16/2019

Does Gender Matter? Towards Fairness in Dialogue Systems

Recently there are increasing concerns about the fairness of Artificial ...
research
09/28/2022

Improving alignment of dialogue agents via targeted human judgements

We present Sparrow, an information-seeking dialogue agent trained to be ...
research
08/15/2023

DiagGPT: An LLM-based Chatbot with Automatic Topic Management for Task-Oriented Dialogue

Large Language Models (LLMs), such as ChatGPT, are becoming increasingly...
research
09/11/2021

Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy

Generating goal-oriented questions in Visual Dialogue tasks is a challen...

Please sign up or login with your details

Forgot password? Click here to reset