MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

05/09/2021
by   Nick Craswell, et al.
0

Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/19/2021

TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime

The TREC Deep Learning (DL) Track studies ad hoc search in the large dat...
research
02/15/2021

Overview of the TREC 2020 deep learning track

This is the second year of the TREC Deep Learning Track, with the goal o...
research
02/25/2021

Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard

Leaderboards are a ubiquitous part of modern research in applied machine...
research
03/17/2020

Overview of the TREC 2019 deep learning track

The Deep Learning Track is a new track for TREC 2019, with the goal of s...
research
04/27/2022

On Survivorship Bias in MS MARCO

Survivorship bias is the tendency to concentrate on the positive outcome...
research
03/25/2019

Trends, Challenges and Adopted Strategies in RoboCup@Home (2019 version)

Scientific competitions are crucial in the field of service robotics. Th...

Please sign up or login with your details

Forgot password? Click here to reset