Meta-learning via Language Model In-context Tuning

by   Yanda Chen, et al.

The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples. To tackle this problem in NLP, we propose in-context tuning, which recasts adaptation and prediction as a simple sequence prediction problem: to form the input sequence, we concatenate the task instruction, the labeled examples, and the target input to predict; to meta-train the model to learn from in-context examples, we fine-tune a pre-trained language model (LM) to predict the target label from the input sequences on a collection of tasks. We benchmark our method on two collections of text classification tasks: LAMA and BinaryClfs. Compared to first-order MAML which adapts the model with gradient descent, our method better leverages the inductive bias of LMs to perform pattern matching, and outperforms MAML by an absolute 6% AUC ROC score on BinaryClfs, with increasing advantage w.r.t. model size. Compared to non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning directly learns to learn from in-context examples. On BinaryClfs, in-context tuning improves the average AUC-ROC score by an absolute 10%, and reduces the variance with respect to example ordering by 6x and example choices by 2x.



page 1

page 2

page 3

page 4


Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

Self-supervised pre-training of transformer models has revolutionized NL...

Meta-Learning with Context-Agnostic Initialisations

Meta-learning approaches have addressed few-shot problems by finding ini...

ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning

Recent research considers few-shot intent detection as a meta-learning p...

MetaICL: Learning to Learn In Context

We introduce MetaICL (Meta-training for In-Context Learning), a new meta...

Meta-learning for Few-shot Natural Language Processing: A Survey

Few-shot natural language processing (NLP) refers to NLP tasks that are ...

Adaptable Text Matching via Meta-Weight Regulator

Neural text matching models have been used in a range of applications su...

GPTs at Factify 2022: Prompt Aided Fact-Verification

One of the most pressing societal issues is the fight against false news...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.