Do Data-based Curricula Work?

12/13/2021
by   Maxim K. Surkov, et al.
0

Current state-of-the-art NLP systems use large neural networks that require lots of computational resources for training. Inspired by human knowledge acquisition, researchers have proposed curriculum learning, - sequencing of tasks (task-based curricula) or ordering and sampling of the datasets (data-based curricula) that facilitate training. This work investigates the benefits of data-based curriculum learning for large modern language models such as BERT and T5. We experiment with various curricula based on a range of complexity measures and different sampling strategies. Extensive experiments on different NLP tasks show that curricula based on various complexity measures rarely has any benefits while random sampling performs either as well or better than curricula.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

12/05/2020

When Do Curricula Work?

Inspired by human learning, researchers have proposed ordering examples ...
05/12/2022

Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?

In recent years, deep neural language models have made strong progress i...
11/14/2021

Curriculum Learning for Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) is a task where an agent navigates ...
03/24/2020

PADS: Policy-Adapted Sampling for Visual Similarity Learning

Learning visual similarity requires to learn relations, typically betwee...
02/08/2022

What are the best systems? New perspectives on NLP Benchmarking

In Machine Learning, a benchmark refers to an ensemble of datasets assoc...
06/05/2019

Energy and Policy Considerations for Deep Learning in NLP

Recent progress in hardware and methodology for training neural networks...
07/12/2018

Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate

Can advances in NLP help advance cognitive modeling? We examine the role...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.