Document Ranking with a Pretrained Sequence-to-Sequence Model

03/14/2020
by   Rodrigo Nogueira, et al.
0

This work proposes a novel adaptation of a pretrained sequence-to-sequence model to the task of document ranking. Our approach is fundamentally different from a commonly-adopted classification-based formulation of ranking, based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words", and how the underlying logits of these target words can be interpreted as relevance probabilities for ranking. On the popular MS MARCO passage ranking task, experimental results show that our approach is at least on par with previous classification-based models and can surpass them with larger, more-recent models. On the test collection from the TREC 2004 Robust Track, we demonstrate a zero-shot transfer-based approach that outperforms previous state-of-the-art models requiring in-dataset cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only model in a data-poor regime (i.e., with few training examples). We investigate this observation further by varying target words to probe the model's use of latent knowledge.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/14/2021

The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models

We propose a design pattern for tackling text ranking problems, dubbed "...
research
12/15/2020

Traditional IR rivals neural models on the MS MARCO Document Ranking Leaderboard

This short document describes a traditional IR system that achieved MRR@...
research
03/21/2022

AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization

Like most natural language understanding and generation tasks, state-of-...
research
10/12/2022

RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses

Recently, substantial progress has been made in text ranking based on pr...
research
03/25/2023

Indian Language Summarization using Pretrained Sequence-to-Sequence Models

The ILSUM shared task focuses on text summarization for two major Indian...
research
12/19/2022

Multilingual Sequence-to-Sequence Models for Hebrew NLP

Recent work attributes progress in NLP to large language models (LMs) wi...
research
08/21/2022

Transfer Ranking in Finance: Applications to Cross-Sectional Momentum with Data Scarcity

Cross-sectional strategies are a classical and popular trading style, wi...

Please sign up or login with your details

Forgot password? Click here to reset