Zero-shot Learning by Generating Task-specific Adapters

01/02/2021
by   Qinyuan Ye, et al.
0

Pre-trained text-to-text transformers achieve impressive performance across a wide range of NLP tasks, and they naturally support zero-shot learning (ZSL) by using the task description as prompt in the input. However, this approach has potential limitations, as it learns from input-output pairs at instance level, instead of learning to solve tasks at task level. Alternatively, applying existing ZSL methods to text-to-text transformers is non-trivial due to their text generation objective and huge size. To address these issues, we introduce Hypter, a framework that improves zero-shot transferability by training a hypernetwork to generate task-specific adapters from task descriptions. This formulation enables learning at task level, and greatly reduces the number of parameters by using light-weight adapters. Experiments on two datasets demonstrate Hypter improves upon fine-tuning baselines.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/23/2021

Zero-Shot Information Extraction as a Unified Text-to-Triple Translation

We cast a suite of information extraction tasks into a text-to-triple tr...
research
10/22/2022

ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback

Recently, dataset-generation-based zero-shot learning has shown promisin...
research
10/05/2020

KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation

Data-to-text generation has recently attracted substantial interests due...
research
07/16/2023

Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training

We introduce DualMind, a generalist agent designed to tackle various dec...
research
04/13/2022

Impossible Triangle: What's Next for Pre-trained Language Models?

Recent development of large-scale pre-trained language models (PLM) have...
research
08/21/2023

Dynamic Strategy Chain: Dynamic Zero-Shot CoT for Long Mental Health Support Generation

Long counseling Text Generation for Mental health support (LTGM), an inn...
research
05/21/2023

Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers

This paper explores the effectiveness of model-generated signals in impr...

Please sign up or login with your details

Forgot password? Click here to reset