How Many Data Samples is an Additional Instruction Worth?

03/17/2022
by   Ravsehaj Singh Puri, et al.
4

Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state of the art task specific models. Conventional approaches to improve model performance via creating large datasets with lots of task instances or architectural/training changes in model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augumentation helpful? We augment a subset of tasks in the expanded version of NATURAL INSTRUCTIONS with additional instructions and find that these significantly improve model performance (up to 35 low-data regime. Our results indicate that an additional instruction can be equivalent to  200 data samples on average across tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/20/2023

Evaluating the Zero-shot Robustness of Instruction-tuned Language Models

Instruction fine-tuning has recently emerged as a promising approach for...
research
03/18/2023

Is Prompt All You Need? No. A Comprehensive and Broader View of Instruction Learning

Task semantics can be expressed by a set of input-to-output examples or ...
research
05/01/2022

Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions

In recent years, progress in NLU has been driven by benchmarks. These be...
research
01/24/2023

Multitask Instruction-based Prompting for Fallacy Recognition

Fallacies are used as seemingly valid arguments to support a position an...
research
07/01/2023

InstructEval: Systematic Evaluation of Instruction Selection Methods

In-context learning (ICL) performs tasks by prompting a large language m...
research
04/19/2022

What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment

The instruction learning paradigm – where a model learns to perform new ...
research
09/21/2023

A Computational Analysis of Vagueness in Revisions of Instructional Texts

WikiHow is an open-domain repository of instructional articles for a var...

Please sign up or login with your details

Forgot password? Click here to reset