What's in a Measurement? Using GPT-3 on SemEval 2021 Task 8 – MeasEval

06/28/2021
by   Curt Kohler, et al.
0

In the summer of 2020 OpenAI released its GPT-3 autoregressive language model to much fanfare. While the model has shown promise on tasks in several areas, it has not always been clear when the results were cherry-picked or when they were the unvarnished output. We were particularly interested in what benefits GPT-3 could bring to the SemEval 2021 MeasEval task - identifying measurements and their associated attributes in scientific literature. We had already experimented with multi-turn questions answering as a solution to this task. We wanted to see if we could use GPT-3's few-shot learning capabilities to more easily develop a solution that would have better performance than our prior work. Unfortunately, we have not been successful in that effort. This paper discusses the approach we used, challenges we encountered, and results we observed. Some of the problems we encountered were simply due to the state of the art. For example, the limits on the size of the prompt and answer limited the amount of the training signal that could be offered. Others are more fundamental. We are unaware of generative models that excel in retaining factual information. Also, the impact of changes in the prompts is unpredictable, making it hard to reliably improve performance.

READ FULL TEXT
research
04/05/2022

PaLM: Scaling Language Modeling with Pathways

Large language models have been shown to achieve remarkable performance ...
research
12/17/2020

On Episodes, Prototypical Networks, and Few-shot Learning

Episodic learning is a popular practice among researchers and practition...
research
06/02/2022

Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code

Few-shot learning with large-scale, pre-trained language models is a pow...
research
07/09/2022

Few-shot training LLMs for project-specific code-summarization

Very large language models (LLMs), such as GPT-3 and Codex have achieved...
research
09/16/2021

Reframing Instructional Prompts to GPTk's Language

How can model designers turn task instructions into effective prompts fo...
research
01/11/2022

HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning

In this work we propose a HyperTransformer, a transformer-based model fo...

Please sign up or login with your details

Forgot password? Click here to reset