C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

04/25/2022
by   Eugene Yang, et al.
3

Pretrained language models have improved effectiveness on numerous tasks, including ad-hoc retrieval. Recent work has shown that continuing to pretrain a language model with auxiliary objectives before fine-tuning on the retrieval task can further improve retrieval effectiveness. Unlike monolingual retrieval, designing an appropriate auxiliary task for cross-language mappings is challenging. To address this challenge, we use comparable Wikipedia articles in different languages to further pretrain off-the-shelf multilingual pretrained models before fine-tuning on the retrieval task. We show that our approach yields improvements in retrieval effectiveness.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/20/2022

Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters

A popular approach to creating a zero-shot cross-language retrieval mode...
research
04/20/2021

B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval

Pre-training and fine-tuning have achieved remarkable success in many do...
research
08/20/2021

Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need

Designing pre-training objectives that more closely resemble the downstr...
research
01/29/2023

Distilling Internet-Scale Vision-Language Models into Embodied Agents

Instruction-following agents must ground language into their observation...
research
09/19/2021

MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

Recent work indicated that pretrained language models (PLMs) such as BER...
research
09/03/2022

Multilingual ColBERT-X

ColBERT-X is a dense retrieval model for Cross Language Information Retr...
research
11/17/2022

Data-Efficient Autoregressive Document Retrieval for Fact Verification

Document retrieval is a core component of many knowledge-intensive natur...

Please sign up or login with your details

Forgot password? Click here to reset