A Few More Examples May Be Worth Billions of Parameters

by   Yuval Kirstain, et al.

We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task's format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often "worth" billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.



There are no comments yet.


page 1

page 4

page 5

page 6


Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering

Video question answering has recently received a lot of attention from m...

Ensembling Strategies for Answering Natural Questions

Many of the top question answering systems today utilize ensembling to i...

Question Answering is a Format; When is it Useful?

Recent years have seen a dramatic expansion of tasks and datasets posed ...

Natural Perturbation for Robust Question Answering

While recent models have achieved human-level scores on many NLP dataset...

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Generative models for open domain question answering have proven to be c...

Talking-Heads Attention

We introduce "talking-heads attention" - a variation on multi-head atten...

Improving Differentiable Neural Computers Through Memory Masking, De-allocation, and Link Distribution Sharpness Control

The Differentiable Neural Computer (DNC) can learn algorithmic and quest...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.