Fine-tuning Pretrained Language Models with Label Attention for Explainable Biomedical Text Classification

08/26/2021
by   Bruce Nguyen, et al.
17

The massive growth of digital biomedical data is making biomedical text indexing and classification increasingly important. Accordingly, previous research has devised numerous deep learning techniques focused on using feedforward, convolutional or recurrent neural architectures. More recently, fine-tuned transformers-based pretrained models (PTMs) have demonstrated superior performance compared to such models in many natural language processing tasks. However, the direct use of PTMs in the biomedical domain is only limited to the target documents, ignoring the rich semantic information in the label descriptions. In this paper, we develop an improved label attention-based architecture to inject semantic label description into the fine-tuning process of PTMs. Results on two public medical datasets show that the proposed fine-tuning scheme outperforms the conventionally fine-tuned PTMs and prior state-of-the-art models. Furthermore, we show that fine-tuning with the label attention mechanism is interpretable in the interpretability study.

READ FULL TEXT
research
12/15/2021

Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing

Motivation: A perennial challenge for biomedical researchers and clinica...
research
09/03/2021

Biomedical Data-to-Text Generation via Fine-Tuning Transformers

Data-to-text (D2T) generation in the biomedical domain is a promising - ...
research
09/14/2023

PerPLM: Personalized Fine-tuning of Pretrained Language Models via Writer-specific Intermediate Learning and Prompts

The meanings of words and phrases depend not only on where they are used...
research
10/21/2019

Constructing Artificial Data for Fine-tuning for Low-Resource Biomedical Text Tagging with Applications in PICO Annotation

Biomedical text tagging systems are plagued by the dearth of labeled tra...
research
12/29/2022

Maximizing Use-Case Specificity through Precision Model Tuning

Language models have become increasingly popular in recent years for tas...
research
09/21/2023

Accelerating Thematic Investment with Prompt Tuned Pretrained Language Models

Prompt Tuning is emerging as a scalable and cost-effective method to fin...
research
08/15/2019

Deep learning for Plankton and Coral Classification

Oceans are the essential lifeblood of the Earth: they provide over 70 ox...

Please sign up or login with your details

Forgot password? Click here to reset