BioBERT: a pre-trained biomedical language representation model for biomedical text mining

by   Jinhyuk Lee, et al.

Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in machine learning, extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, as deep learning models require a large amount of training data, applying deep learning to biomedical text mining is often unsuccessful due to the lack of training data in biomedical fields. Recent researches on training contextualized language representation models on text corpora shed light on the possibility of leveraging a large number of unannotated biomedical text corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain specific language representation model pre-trained on large-scale biomedical corpora. Based on the BERT architecture, BioBERT effectively transfers the knowledge from a large amount of biomedical texts to biomedical text mining models with minimal task-specific architecture modifications. While BERT shows competitive performances with previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (1.86 (3.33 improvement). We make the pre-trained weights of BioBERT freely available at, and the source code for fine-tuning BioBERT available at


page 1

page 2

page 3

page 4


BioBERT: pre-trained biomedical language representation model for biomedical text mining

Biomedical text mining has become more important than ever as the number...

Conceptualized Representation Learning for Chinese Biomedical Text Mining

Biomedical text mining is becoming increasingly important as the number ...

Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts

In recent years, summarizers that incorporate domain knowledge into the ...

Text-guided Foundation Model Adaptation for Pathological Image Classification

The recent surge of foundation models in computer vision and natural lan...

Transfer learning for biomedical named entity recognition with neural networks.

Motivation The explosive increase of biomedical literature has made i...

Using Distributed Representations to Disambiguate Biomedical and Clinical Concepts

In this paper, we report a knowledge-based method for Word Sense Disambi...

DI2: prior-free and multi-item discretization ofbiomedical data and its applications

Motivation: A considerable number of data mining approaches for biomedic...

Please sign up or login with your details

Forgot password? Click here to reset