An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model

09/10/2021
by   Kijong Han, et al.
0

Multi-turn response selection models have recently shown comparable performance to humans in several benchmark datasets. However, in the real environment, these models often have weaknesses, such as making incorrect predictions based heavily on superficial patterns without a comprehensive understanding of the context. For example, these models often give a high score to the wrong response candidate containing several keywords related to the context but using the inconsistent tense. In this study, we analyze the weaknesses of the open-domain Korean Multi-turn response selection models and publish an adversarial dataset to evaluate these weaknesses. We also suggest a strategy to build a robust model in this adversarial environment.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/10/2018

Improving Response Selection in Multi-turn Dialogue Systems

Building systems that can communicate with humans is a core problem in A...
research
11/19/2021

Small Changes Make Big Differences: Improving Multi-turn Response Selection in Dialogue Systems via Fine-Grained Contrastive Learning

Retrieve-based dialogue response selection aims to find a proper respons...
research
04/07/2020

Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots

In this paper, we study the problem of employing pre-trained language mo...
research
02/15/2018

Improving Retrieval Modeling Using Cross Convolution Networks And Multi Frequency Word Embedding

To build a satisfying chatbot that has the ability of managing a goal-or...
research
04/06/2020

Grayscale Data Construction and Multi-Level Ranking Objective for Dialogue Response Selection

Response selection plays a vital role in building retrieval-based conver...
research
04/04/2020

Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks

We study multi-turn response generation for open-domain dialogues. The e...
research
02/18/2020

Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting

Open-domain retrieval-based dialogue systems require a considerable amou...

Please sign up or login with your details

Forgot password? Click here to reset