Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

03/16/2022
by   Bernal Jimenez Gutierrez, et al.
0

The strong few-shot in-context learning capability of large pre-trained language models (PLMs) such as GPT-3 is highly appealing for biomedical applications where data annotation is particularly costly. In this paper, we present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two highly representative biomedical information extraction tasks, named entity recognition and relation extraction. We follow the true few-shot setting to avoid overestimating models' few-shot performance by model selection over a large validation set. We also optimize GPT-3's performance with known techniques such as contextual calibration and dynamic in-context example retrieval. However, our results show that GPT-3 still significantly underperforms compared with simply fine-tuning a smaller PLM using the same small training set. Moreover, what is equally important for practical applications is that adding more labeled data would reliably yield an improvement in model performance. While that is the case when fine-tuning small PLMs, GPT-3's performance barely improves when adding more data. In-depth analyses further reveal issues of the in-context learning setting that may be detrimental to information extraction tasks in general. Given the high cost of experimenting with GPT-3, we hope our study provides guidance for biomedical researchers and practitioners towards more promising directions such as fine-tuning GPT-3 or small PLMs.

READ FULL TEXT
research
04/21/2022

Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction

Relation extraction is a core problem for natural language processing in...
research
11/01/2020

Investigation of BERT Model on Biomedical Relation Extraction Based on Revised Fine-tuning Mechanism

With the explosive growth of biomedical literature, designing automatic ...
research
09/14/2022

Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words

Prompt-based fine-tuning for pre-trained models has proven effective for...
research
04/17/2022

Pathologies of Pre-trained Language Models in Few-shot Fine-tuning

Although adapting pre-trained language models with few examples has show...
research
06/13/2023

Few-shot learning for sentence pair classification and its applications in software engineering

Few-shot learning-the ability to train models with access to limited dat...
research
10/12/2021

LiST: Lite Self-training Makes Efficient Few-shot Learners

We present a new method LiST for efficient fine-tuning of large pre-trai...
research
03/06/2023

Dynamic Prompting: A Unified Framework for Prompt Tuning

It has been demonstrated that prompt tuning is highly effective in effic...

Please sign up or login with your details

Forgot password? Click here to reset