Deep learning with sentence embeddings pre-trained on biomedical corpora improves the performance of finding similar sentences in electronic medical records

by   Qingyu Chen, et al.

Capturing sentence semantics plays a vital role in a range of text mining applications. Despite continuous efforts on the development of related datasets and models in the general domain, both datasets and models are limited in biomedical and clinical domains. The BioCreative/OHNLP organizers have made the first attempt to annotate 1,068 sentence pairs from clinical notes and have called for a community effort to tackle the Semantic Textual Similarity (BioCreative/OHNLP STS) challenge. We developed models using traditional machine learning and deep learning approaches. For the post challenge, we focus on two models: the Random Forest and the Encoder Network. We applied sentence embeddings pre-trained on PubMed abstracts and MIMIC-III clinical notes and updated the Random Forest and the Encoder Network accordingly. The official results demonstrated our best submission was the ensemble of eight models. It achieved a Person correlation coefficient of 0.8328, the highest performance among 13 submissions from 4 teams. For the post challenge, the performance of both Random Forest and the Encoder Network was improved; in particular, the correlation of the Encoder Network was improved by 13 task, no end-to-end deep learning models had better performance than machine learning models that take manually-crafted features. In contrast, with the sentence embeddings pre-trained on biomedical corpora, the Encoder Network now achieves a correlation of 0.84, which is higher than the original best model. The ensembled model taking the improved versions of the Random Forest and Encoder Network as inputs further increased performance to 0.8528. Deep learning models with sentence embeddings pre-trained on biomedical corpora achieve the highest performance on the test set.


page 3

page 8


BioSentVec: creating sentence embeddings for biomedical texts

Sentence embeddings have become an essential part of today's natural lan...

Predicting Heart Failure Readmission from Clinical Notes Using Deep Learning

Heart failure hospitalization is a severe burden on healthcare. How to p...

Quantification of BERT Diagnosis Generalizability Across Medical Specialties Using Semantic Dataset Distance

Deep learning models in healthcare may fail to generalize on data from u...

DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

In recent years, pre-trained language models (PLMs) achieve the best per...

TF-IDF vs Word Embeddings for Morbidity Identification in Clinical Notes: An Initial Study

Today, we are seeing an ever-increasing number of clinical notes that co...

Detecting state of aggression in sentences using CNN

In this article we study verbal expression of aggression and its detecti...

Literature Triage on Genomic Variation Publications by Knowledge-enhanced Multi-channel CNN

Background: To investigate the correlation between genomic variation and...

Please sign up or login with your details

Forgot password? Click here to reset