Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search

05/25/2019
by   Pepa Atanasova, et al.
0

In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links" metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, along came chatbots, conversational systems, and messaging platforms, where the user needs could be better served with the system asking follow-up questions in order to better understand the user's intent. While typically a user would expect a single response at any utterance, a system could also return multiple options for the user to select from, based on different system understandings of the user's intent. However, this possibility should not be overused, as this practice could confuse and/or annoy the user. How to produce good variable-length lists, given the conflicting objectives of staying short while maximizing the likelihood of having a correct answer included in the list, is an underexplored problem. It is also unclear how to evaluate a system that tries to do that. Here we aim to bridge this gap. In particular, we define some necessary and some optional properties that an evaluation measure fit for this purpose should have. We further show that existing evaluation measures from the IR tradition are not entirely suitable for this setup, and we propose novel evaluation measures that address it satisfactorily.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/28/2021

Multi-stage Clarification in Conversational AI: The case of Question-Answering Dialogue Systems

Clarification resolution plays an important role in various information ...
research
07/12/2021

Asking Clarifying Questions Based on Negative Feedback in Conversational Search

Users often need to look through multiple search result pages or reformu...
research
06/20/2020

IQA: Interactive Query Construction in Semantic Question Answering Systems

Semantic Question Answering (SQA) systems automatically interpret user q...
research
05/11/2021

Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphs

The rise of personal assistants has made conversational question answeri...
research
04/15/2023

Intent-aware Ranking Ensemble for Personalized Recommendation

Ranking ensemble is a critical component in real recommender systems. Wh...
research
06/11/2018

Prosody Modifications for Question-Answering in Voice-Only Settings

Many popular form factors of digital assistant---such as Amazon Echo, Ap...
research
08/27/2021

A framework for massive scale personalized promotion

Technology companies building consumer-facing platforms may have access ...

Please sign up or login with your details

Forgot password? Click here to reset