Challenges for Measuring Usefulness of Interactive IR Systems with Log-based Approaches

09/07/2018
by   Daniel Hienert, et al.
0

The usefulness evaluation model proposed by Cole et al. in 2009 [2] focuses on the evaluation of interactive IR systems by their support towards the user's overall goal, sub goals and tasks. This is a more human focus of the IR evaluation process than with classical TREC-oriented studies and gives a more holistic view on the IR evaluation process. However, yet there is no formal framework how the usefulness model can be operationalized. Additionally, a lot of information needed for the operationalization is only available in explicit user studies where for example the overall goal and the tasks are prompted from the users or are predefined. Measuring the usefulness of IR systems outside the laboratory is a challenging task as most often only log data of user interaction is available. But, an operationalization of the usefulness model based on interaction data could be applied to diverse systems and evaluation results would be comparable. In this article we discuss the challenges for measuring the usefulness of IIR systems with log-based approaches.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/21/2018

A Usefulness-based Approach for Measuring the Local and Global Effect of IIR Services

In Interactive Information Retrieval (IIR) different services such as se...
research
01/31/2019

An InfoVis Tool for Interactive Component-Based Evaluation

In this paper, we present an InfoVis tool based on Sankey diagrams for t...
research
02/05/2019

Mobile Information Retrieval

Mobile Information Retrieval (Mobile IR) is a relatively recent branch o...
research
12/02/2022

Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking

Neural information retrieval (IR) systems have progressed rapidly in rec...
research
09/07/2018

Data Requirements for Evaluation of Personalization of Information Retrieval - A Position Paper

Two key, but usually ignored, issues for the evaluation of methods of pe...
research
10/31/2018

A Process-driven View on Summative Evaluation of Visual Analytics Solutions

Many evaluation methods have been applied to assess the usefulness of vi...
research
09/05/2022

Using Consensual Biterms from Text Structures of Requirements and Code to Improve IR-Based Traceability Recovery

Traceability approves trace links among software artifacts based on whet...

Please sign up or login with your details

Forgot password? Click here to reset