Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning

by   Gordon Buck, et al.

Word embedding learning methods require a large number of occurrences of a word to accurately learn its embedding. However, out-of-vocabulary (OOV) words which do not appear in the training corpus emerge frequently in the smaller downstream data. Recent work formulated OOV embedding learning as a few-shot regression problem and demonstrated that meta-learning can improve results obtained. However, the algorithm used, model-agnostic meta-learning (MAML) is known to be unstable and perform worse when a large number of gradient steps are used for parameter updates. In this work, we propose the use of Leap, a meta-learning algorithm which leverages the entire trajectory of the learning process instead of just the beginning and the end points, and thus ameliorates these two issues. In our experiments on a benchmark OOV embedding learning dataset and in an extrinsic evaluation, Leap performs comparably or better than MAML. We go on to examine which contexts are most beneficial to learn an OOV embedding from, and propose that the choice of contexts may matter more than the meta-learning employed.



There are no comments yet.


page 1

page 2

page 3

page 4


Few-Shot Representation Learning for Out-Of-Vocabulary Words

Existing approaches for learning word embeddings often assume there are ...

A Concise Review of Recent Few-shot Meta-learning Methods

Few-shot meta-learning has been recently reviving with expectations to m...

Lifelong Domain Word Embedding via Meta-Learning

Learning high-quality domain word embeddings is important for achieving ...

A contrastive rule for meta-learning

Meta-learning algorithms leverage regularities that are present on a set...

Meta-Learning with Variational Semantic Memory for Word Sense Disambiguation

A critical challenge faced by supervised word sense disambiguation (WSD)...

Warm Up Cold-start Advertisements: Improving CTR Predictions via Learning to Learn ID Embeddings

Click-through rate (CTR) prediction has been one of the most central pro...

Learning not to learn: Nature versus nurture in silico

Animals are equipped with a rich innate repertoire of sensory, behaviora...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.