DeepAI AI Chat
Log In Sign Up

Stress Test Evaluation of Biomedical Word Embeddings

07/24/2021
by   Vladimir Araujo, et al.
Elsevier
Pontificia Universidad Católica de Chile
17

The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research on quantifying their behavior under severe "stress" scenarios. In this work, we systematically evaluate three language models with adversarial examples – automatically constructed tests that allow us to examine how robust the models are. We propose two types of stress scenarios focused on the biomedical named entity recognition (NER) task, one inspired by spelling errors and another based on the use of synonyms for medical terms. Our experiments with three benchmarks show that the performance of the original models decreases considerably, in addition to revealing their weaknesses and strengths. Finally, we show that adversarial training causes the models to improve their robustness and even to exceed the original performance in some cases.

READ FULL TEXT
04/23/2020

On Adversarial Examples for Biomedical NLP Tasks

The success of pre-trained word embeddings has motivated its use in task...
04/07/2020

Inexpensive Domain Adaptation of Pretrained Language Models: A Case Study on Biomedical Named Entity Recognition

Domain adaptation of Pretrained Language Models (PTLMs) is typically ach...
09/23/2021

Breaking BERT: Understanding its Vulnerabilities for Biomedical Named Entity Recognition through Adversarial Attack

Biomedical named entity recognition (NER) is a key task in the extractio...
04/03/2019

Probing Biomedical Embeddings from Language Models

Contextualized word embeddings derived from pre-trained language models ...
08/13/2019

BioFLAIR: Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks

Biomedical Named Entity Recognition (NER) is a challenging problem in bi...
11/16/2021

Improving the robustness and accuracy of biomedical language models through adversarial training

Deep transformer neural network models have improved the predictive accu...
11/12/2018

Bio-YODIE: A Named Entity Linking System for Biomedical Text

Ever-expanding volumes of biomedical text require automated semantic ann...