Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning

by   Daniel Grießhaber, et al.

Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks. However, only a little research has explored the suitability of this approach in low resource settings with less than 1,000 training data points. In this work, we explore fine-tuning methods of BERT – a pre-trained Transformer based language model – by utilizing pool-based active learning to speed up training while keeping the cost of labeling new data constant. Our experimental results on the GLUE data set show an advantage in model performance by maximizing the approximate knowledge gain of the model when querying from the pool of unlabeled data. Finally, we demonstrate and analyze the benefits of freezing layers of the language model during fine-tuning to reduce the number of trainable parameters, making it more suitable for low-resource settings.


page 2

page 6


Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings

Pre-trained language models (PLMs) have ignited a surge in demand for ef...

Neural Grapheme-to-Phoneme Conversion with Pre-trained Grapheme Models

Neural network models have achieved state-of-the-art performance on grap...

Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding

End-to-end spoken language understanding (SLU) systems benefit from pret...

Simple Recurrence Improves Masked Language Models

In this work, we explore whether modeling recurrence into the Transforme...

Revisiting Sample Size Determination in Natural Language Understanding

Knowing exactly how many data points need to be labeled to achieve a cer...

Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning

Pre-trained Language Models (PLMs) have achieved remarkable performance ...

Please sign up or login with your details

Forgot password? Click here to reset