DeepAI AI Chat
Log In Sign Up

Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU

by   Fenia Christopoulou, et al.
HUAWEI Technologies Co., Ltd.

Curriculum Learning (CL) is a technique of training models via ranking examples in a typically increasing difficulty trend with the aim of accelerating convergence and improving generalisability. Current approaches for Natural Language Understanding (NLU) tasks use CL to improve in-distribution data performance often via heuristic-oriented or task-agnostic difficulties. In this work, instead, we employ CL for NLU by taking advantage of training dynamics as difficulty metrics, i.e., statistics that measure the behavior of the model at hand on specific task-data instances during training and propose modifications of existing CL schedulers based on these statistics. Differently from existing works, we focus on evaluating models on in-distribution (ID), out-of-distribution (OOD) as well as zero-shot (ZS) cross-lingual transfer datasets. We show across several NLU tasks that CL with training dynamics can result in better performance mostly on zero-shot cross-lingual transfer and OOD settings with improvements up by 8.5 indicate that training dynamics can lead to better performing models with smoother training compared to other difficulty metrics while being 20 on average. In addition, through analysis we shed light on the correlations of task-specific versus task-agnostic metrics.


page 1

page 2

page 3

page 4


English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

Intermediate-task training has been shown to substantially improve pretr...

Task-Specific Pre-Training and Cross Lingual Transfer for Code-Switched Data

Using task-specific pre-training and leveraging cross-lingual transfer a...

CrossAligner Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding

Task-oriented personal assistants enable people to interact with a host ...

Worst-Case-Aware Curriculum Learning for Zero and Few Shot Transfer

Multi-task transfer learning based on pre-trained language encoders achi...

Language Model Priming for Cross-Lingual Event Extraction

We present a novel, language-agnostic approach to "priming" language mod...

Meta-Learning with MAML on Trees

In meta-learning, the knowledge learned from previous tasks is transferr...

The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains

Recent model pruning methods have demonstrated the ability to remove red...