Are Neural Ranking Models Robust?

08/11/2021
by   Chen Wu, et al.
0

Recently, we have witnessed the bloom of neural ranking models in the information retrieval (IR) field. So far, much effort has been devoted to developing effective neural ranking models that can generalize well on new data. There has been less attention paid to the robustness perspective. Unlike the effectiveness which is about the average performance of a system under normal purpose, robustness cares more about the system performance in the worst case or under malicious operations instead. When a new technique enters into the real-world application, it is critical to know not only how it works in average, but also how would it behave in abnormal situations. So we raise the question in this work: Are neural ranking models robust? To answer this question, firstly, we need to clarify what we refer to when we talk about the robustness of ranking models in IR. We show that robustness is actually a multi-dimensional concept and there are three ways to define it in IR: 1) The performance variance under the independent and identically distributed (I.I.D.) setting; 2) The out-of-distribution (OOD) generalizability; and 3) The defensive ability against adversarial operations. The latter two definitions can be further specified into two different perspectives respectively, leading to 5 robustness tasks in total. Based on this taxonomy, we build corresponding benchmark datasets, design empirical experiments, and systematically analyze the robustness of several representative neural ranking models against traditional probabilistic ranking models and learning-to-rank (LTR) models. The empirical results show that there is no simple answer to our question. While neural ranking models are less robust against other IR models in most cases, some of them can still win 1 out of 5 tasks. This is the first comprehensive study on the robustness of neural ranking models.

READ FULL TEXT

page 9

page 16

page 19

page 26

research
06/22/2023

On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspective

Recently, we have witnessed generative retrieval increasingly gaining at...
research
03/16/2019

A Deep Look into Neural Ranking Models for Information Retrieval

Ranking models lie at the heart of research on information retrieval (IR...
research
06/15/2021

Towards Axiomatic Explanations for Neural Ranking Models

Recently, neural networks have been successfully employed to improve upo...
research
01/10/2022

Continual Learning of Long Topic Sequences in Neural Information Retrieval

In information retrieval (IR) systems, trends and users' interests may c...
research
06/12/2018

Ranking Robustness Under Adversarial Document Manipulations

For many queries in the Web retrieval setting there is an on-going ranki...
research
04/17/2023

Metric-agnostic Ranking Optimization

Ranking is at the core of Information Retrieval. Classic ranking optim...
research
03/12/2021

Quantitative robustness of instance ranking problems

Instance ranking problems intend to recover the true ordering of the ins...

Please sign up or login with your details

Forgot password? Click here to reset