Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency

06/04/2021
by   Zekang Li, et al.
0

A good open-domain chatbot should avoid presenting contradictory responses about facts or opinions in a conversational session, known as its consistency capacity. However, evaluating the consistency capacity of a chatbot is still challenging. Employing human judges to interact with chatbots on purpose to check their capacities is costly and low-efficient, and difficult to get rid of subjective bias. In this paper, we propose the Addressing Inquiries about History (AIH), an efficient and practical framework for the consistency evaluation. At the conversation stage, AIH attempts to address appropriate inquiries about the dialogue history to induce the chatbot to redeclare the historical facts or opinions. We carry out the conversation between chatbots, which is more efficient than the human-bot interaction and can also alleviate the subjective bias. In this way, we manage to rapidly obtain a dialog session that contains responses with high contradiction possibilities. At the contradiction recognition stage, we can either employ human judges or a natural language inference (NLI) model to recognize whether the answers to the inquiries are contradictory with history. Finally, we are able to rank chatbots according to the contradiction statistics. Experiments on open-domain chatbots show that our approach can efficiently and reliably assess the consistency capacity of chatbots and achieve a high ranking correlation with the human evaluation. We release the framework and hope to help improve the consistency capacity of chatbots. [<https://github.com/ictnlp/AIH>]

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/04/2021

Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances

Nowadays, open-domain dialogue models can generate acceptable responses ...
research
09/23/2021

Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Consistency Identification has obtained remarkable success on open-domai...
research
07/14/2023

Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots

Recent advances in natural language processing and machine learning have...
research
06/28/2022

SINC: Service Information Augmented Open-Domain Conversation

Generative open-domain dialogue systems can benefit from external knowle...
research
05/13/2019

Challenges in Building Intelligent Open-domain Dialog Systems

There is a resurgent interest in developing intelligent open-domain dial...
research
12/02/2021

Evaluator for Emotionally Consistent Chatbots

One challenge for evaluating current sequence- or dialogue-level chatbot...

Please sign up or login with your details

Forgot password? Click here to reset