Minimax and Neyman-Pearson Meta-Learning for Outlier Languages

06/02/2021
by   Edoardo Maria Ponti, et al.
0

Model-agnostic meta-learning (MAML) has been recently put forth as a strategy to learn resource-poor languages in a sample-efficient fashion. Nevertheless, the properties of these languages are often not well represented by those available during training. Hence, we argue that the i.i.d. assumption ingrained in MAML makes it ill-suited for cross-lingual NLP. In fact, under a decision-theoretic framework, MAML can be interpreted as minimising the expected risk across training languages (with a uniform prior), which is known as Bayes criterion. To increase its robustness to outlier languages, we create two variants of MAML based on alternative criteria: Minimax MAML reduces the maximum risk across languages, while Neyman-Pearson MAML constrains the risk in each language to a maximum threshold. Both criteria constitute fully differentiable two-player games. In light of this, we propose a new adaptive optimiser solving for a local approximation to their Nash equilibrium. We evaluate both model variants on two popular NLP tasks, part-of-speech tagging and question answering. We report gains for their average and minimum performance across low-resource languages in zero- and few-shot settings, compared to joint multi-source transfer and vanilla MAML.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/19/2022

Meta-X_NLG: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation

Recently, the NLP community has witnessed a rapid advancement in multili...
research
11/10/2021

Cross-lingual Adaption Model-Agnostic Meta-Learning for Natural Language Understanding

Meta learning with auxiliary languages has demonstrated promising improv...
research
03/05/2020

Zero-Shot Cross-Lingual Transfer with Meta Learning

Learning what to share between tasks has been a topic of high importance...
research
01/25/2021

Meta-Learning for Effective Multi-task and Multilingual Modelling

Natural language processing (NLP) tasks (e.g. question-answering in Engl...
research
01/27/2021

Multilingual and cross-lingual document classification: A meta-learning approach

The great majority of languages in the world are considered under-resour...
research
01/30/2020

Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages

Most combinations of NLP tasks and language varieties lack in-domain exa...

Please sign up or login with your details

Forgot password? Click here to reset