Do Data-based Curricula Work?

12/13/2021
by   Maxim K. Surkov, et al.
0

Current state-of-the-art NLP systems use large neural networks that require lots of computational resources for training. Inspired by human knowledge acquisition, researchers have proposed curriculum learning, - sequencing of tasks (task-based curricula) or ordering and sampling of the datasets (data-based curricula) that facilitate training. This work investigates the benefits of data-based curriculum learning for large modern language models such as BERT and T5. We experiment with various curricula based on a range of complexity measures and different sampling strategies. Extensive experiments on different NLP tasks show that curricula based on various complexity measures rarely has any benefits while random sampling performs either as well or better than curricula.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/05/2020

When Do Curricula Work?

Inspired by human learning, researchers have proposed ordering examples ...
research
06/27/2023

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

State-of-the-art supervised NLP models achieve high accuracy but are als...
research
07/14/2023

HuCurl: Human-induced Curriculum Discovery

We introduce the problem of curriculum discovery and describe a curricul...
research
05/12/2022

Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?

In recent years, deep neural language models have made strong progress i...
research
03/24/2020

PADS: Policy-Adapted Sampling for Visual Similarity Learning

Learning visual similarity requires to learn relations, typically betwee...
research
06/05/2019

Energy and Policy Considerations for Deep Learning in NLP

Recent progress in hardware and methodology for training neural networks...
research
02/08/2022

What are the best systems? New perspectives on NLP Benchmarking

In Machine Learning, a benchmark refers to an ensemble of datasets assoc...

Please sign up or login with your details

Forgot password? Click here to reset