Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Setting

09/17/2021
by   Lukas Lange, et al.
0

In this paper, we explore possible improvements of transformer models in a low-resource setting. In particular, we present our approaches to tackle the first two of three subtasks of the MEDDOPROF competition, i.e., the extraction and classification of job expressions in Spanish clinical texts. As neither language nor domain experts, we experiment with the multilingual XLM-R transformer model and tackle these low-resource information extraction tasks as sequence-labeling problems. We explore domain- and language-adaptive pretraining, transfer learning and strategic datasplits to boost the transformer model. Our results show strong improvements using these methods by up to 5.3 F1 points compared to a fine-tuned XLM-R model. Our best models achieve 83.2 and 79.3 F1 for the first two tasks, respectively.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/23/2022

MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning

Transformer language models (TLMs) are critical for most NLP tasks, but ...
research
10/13/2020

Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language

Low-resource languages such as Filipino suffer from data scarcity which ...
research
05/22/2023

Evaluating Prompt-based Question Answering for Object Prediction in the Open Research Knowledge Graph

There have been many recent investigations into prompt-based training of...
research
04/16/2021

To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning

In low-resource settings, model transfer can help to overcome a lack of ...
research
08/29/2023

Improving Neural Ranking Models with Traditional IR Methods

Neural ranking methods based on large transformer models have recently g...
research
02/16/2022

Knowledge Extraction in Low-Resource Scenarios: Survey and Perspective

Knowledge Extraction (KE) which aims to extract structural information f...
research
11/05/2020

Competence-Level Prediction and Resume Job Description Matching Using Context-Aware Transformer Models

This paper presents a comprehensive study on resume classification to re...

Please sign up or login with your details

Forgot password? Click here to reset