Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature

by   Yu Wang, et al.

Information overload is a prevalent challenge in many high-value domains. A prominent case in point is the explosion of the biomedical literature on COVID-19, which swelled to hundreds of thousands of papers in a matter of months. In general, biomedical literature expands by two papers every minute, totalling over a million new papers every year. Search in the biomedical realm, and many other vertical domains is challenging due to the scarcity of direct supervision from click logs. Self-supervised learning has emerged as a promising direction to overcome the annotation bottleneck. We propose a general approach for vertical search based on domain-specific pretraining and present a case study for the biomedical domain. Despite being substantially simpler and not using any relevance labels for training or development, our method performs comparably or better than the best systems in the official TREC-COVID evaluation, a COVID-related biomedical search competition. Using distributed computing in modern cloud infrastructure, our system can scale to tens of millions of articles on PubMed and has been deployed as Microsoft Biomedical Search, a new search experience for biomedical literature:


Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing

Contrastive pretraining on parallel image-text data has attained great s...

LitMC-BERT: transformer-based multi-label classification of biomedical literature with an application on COVID-19 literature curation

The rapid growth of biomedical literature poses a significant challenge ...

Patent Citation Spectroscopy (PCS): Algorithmic retrieval of landmark patents

One essential component in the construction of patent landscapes in biom...

CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search

Neural rankers based on deep pretrained language models (LMs) have been ...

Coreference Resolution for the Biomedical Domain: A Survey

Issues with coreference resolution are one of the most frequently mentio...

PubMed Labs: An experimental platform for improving biomedical literature search

PubMed is a freely accessible system for searching the biomedical litera...

Please sign up or login with your details

Forgot password? Click here to reset