A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

03/04/2021
by   Iurii Mokrii, et al.
0

Due to high annotation costs, making the best use of existing human-created training data is an important research direction. We, therefore, carry out a systematic evaluation of transferability of BERT-based neural ranking models across five English datasets. Previous studies focused primarily on zero-shot and few-shot transfer from a large dataset to a dataset with a small number of queries. In contrast, each of our collections has a substantial number of queries, which enables a full-shot evaluation mode and improves reliability of our results. Furthermore, since source datasets licences often prohibit commercial use, we compare transfer learning to training on pseudo-labels generated by a BM25 scorer. We find that training on pseudo-labels – possibly with subsequent fine-tuning using a modest number of annotated queries – can produce a competitive or better model compared to transfer learning. However, there is a need to improve the stability and/or effectiveness of the few-shot training, which, in some cases, can degrade performance of a pretrained model.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/09/2022

Model-Agnostic Multitask Fine-tuning for Few-shot Vision-Language Transfer Learning

Despite achieving state-of-the-art zero-shot performance, existing visio...
12/12/2021

Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations

Recently, there has been an increasing interest in models that generate ...
04/30/2020

On the Evaluation of Contextual Embeddings for Zero-Shot Cross-Lingual Transfer Learning

Pre-trained multilingual contextual embeddings have demonstrated state-o...
07/30/2019

Zero-shot transfer for implicit discourse relation classification

Automatically classifying the relation between sentences in a discourse ...
02/16/2021

FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary

Current models for Word Sense Disambiguation (WSD) struggle to disambigu...
02/15/2021

Overview of the TREC 2020 deep learning track

This is the second year of the TREC Deep Learning Track, with the goal o...
04/19/2019

An Evaluation of Transfer Learning for Classifying Sales Engagement Emails at Large Scale

This paper conducts an empirical investigation to evaluate transfer lear...