DeepAI AI Chat
Log In Sign Up

How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning

by   Lu Li, et al.

Automatic dialogue evaluation plays a crucial role in open-domain dialogue research. Previous works train neural networks with limited annotation for conducting automatic dialogue evaluation, which would naturally affect the evaluation fairness as dialogue systems close to the scope of training corpus would have more preference than the other ones. In this paper, we study alleviating this problem from the perspective of continual learning: given an existing neural dialogue evaluator and the next system to be evaluated, we fine-tune the learned neural evaluator by selectively forgetting/updating its parameters, to jointly fit dialogue systems have been and will be evaluated. Our motivation is to seek for a lifelong and low-cost automatic evaluation for dialogue systems, rather than to reconstruct the evaluator over and over again. Experimental results show that our continual evaluator achieves comparable performance with reconstructing new evaluators, while requires significantly lower resources.


page 1

page 2

page 3

page 4


Continual Learning in Task-Oriented Dialogue Systems

Continual learning in task-oriented dialogue systems can allow us to add...

Continual Dialogue State Tracking via Example-Guided Question Answering

Dialogue systems are frequently updated to accommodate new services, but...

Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning

Continual learning is one of the key components of human learning and a ...

Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems

Automation of dialogue system evaluation is a driving force for the effi...

Lifelong and Continual Learning Dialogue Systems

Dialogue systems, commonly known as chatbots, have gained escalating pop...

Automatic Evaluation and Moderation of Open-domain Dialogue Systems

The development of Open-Domain Dialogue Systems (ODS)is a trending topic...