When Do Curricula Work?

12/05/2020
by   Xiaoxia Wu, et al.
0

Inspired by human learning, researchers have proposed ordering examples during training based on their difficulty. Both curriculum learning, exposing a network to easier examples early in training, and anti-curriculum learning, showing the most difficult examples first, have been suggested as improvements to the standard i.i.d. training. In this work, we set out to investigate the relative benefits of ordered learning. We first investigate the implicit curricula resulting from architectural and optimization bias and find that samples are learned in a highly consistent order. Next, to quantify the benefit of explicit curricula, we conduct extensive experiments over thousands of orderings spanning three kinds of learning: curriculum, anti-curriculum, and random-curriculum – in which the size of the training dataset is dynamically increased over time, but the examples are randomly ordered. We find that for standard benchmark datasets, curricula have only marginal benefits, and that randomly ordered samples perform as well or better than curricula and anti-curricula, suggesting that any benefit is entirely due to the dynamic training set size. Inspired by common use cases of curriculum learning in practice, we investigate the role of limited training time budget and noisy data in the success of curriculum learning. Our experiments demonstrate that curriculum, but not anti-curriculum can indeed improve the performance either with limited training time budget or in existence of noisy data.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 5

page 19

page 20

page 22

page 23

page 24

page 25

02/27/2021

Statistical Measures For Defining Curriculum Scoring Function

Curriculum learning is a training strategy that sorts the training examp...
12/13/2021

Do Data-based Curricula Work?

Current state-of-the-art NLP systems use large neural networks that requ...
05/10/2020

A SentiWordNet Strategy for Curriculum Learning in Sentiment Analysis

Curriculum Learning (CL) is the idea that learning on a training set seq...
07/19/2020

Learning Error-Driven Curriculum for Crowd Counting

Density regression has been widely employed in crowd counting. However, ...
06/16/2021

TSO: Curriculum Generation using continuous optimization

The training of deep learning models poses vast challenges of including ...
10/29/2019

Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

We develop techniques to quantify the degree to which a given (training ...
12/18/2019

Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking

Neural ranking models are traditionally trained on a series of random ba...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.