Evaluating Conversational Recommender Systems via User Simulation

06/15/2020
by   Shuo Zhang, et al.
0

Conversational information access is an emerging research area. Currently, human evaluation is used for end-to-end system evaluation, which is both very time and resource intensive at scale, and thus becomes a bottleneck of progress. As an alternative, we propose automated evaluation by means of simulating users. Our user simulator aims to generate responses that a real human would give by considering both individual preferences and the general flow of interaction with the system. We evaluate our simulation approach on an item recommendation task by comparing three existing conversational recommender systems. We show that preference modeling and task-specific interaction models both contribute to more realistic simulations, and can help achieve high correlation between automatic evaluation measures and manual human assessments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/08/2022

Towards Fair Conversational Recommender Systems

Conversational recommender systems have demonstrated great success. They...
research
06/14/2023

User Simulation for Evaluating Information Access Systems

Information access systems, such as search engines, recommender systems,...
research
05/03/2022

Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems

User simulation has been a cost-effective technique for evaluating conve...
research
09/07/2022

INFACT: An Online Human Evaluation Framework for Conversational Recommendation

Conversational recommender systems (CRS) are interactive agents that sup...
research
04/30/2023

Contextual Response Interpretation for Automated Structured Interviews: A Case Study in Market Research

Structured interviews are used in many settings, importantly in market r...
research
09/08/2020

IAI MovieBot: A Conversational Movie Recommender System

Conversational recommender systems support users in accomplishing recomm...
research
01/13/2023

UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems

We present an extensible user simulation toolkit to facilitate automatic...

Please sign up or login with your details

Forgot password? Click here to reset