Empirical Gaussian priors for cross-lingual transfer learning

01/09/2016
by   Anders Søgaard, et al.
0

Sequence model learning algorithms typically maximize log-likelihood minus the norm of the model (or minimize Hamming loss + norm). In cross-lingual part-of-speech (POS) tagging, our target language training data consists of sequences of sentences with word-by-word labels projected from translations in k languages for which we have labeled data, via word alignments. Our training data is therefore very noisy, and if Rademacher complexity is high, learning algorithms are prone to overfit. Norm-based regularization assumes a constant width and zero mean prior. We instead propose to use the k source language models to estimate the parameters of a Gaussian prior for learning new POS taggers. This leads to significantly better performance in multi-source transfer set-ups. We also present a drop-out version that injects (empirical) Gaussian noise during online learning. Finally, we note that using empirical Gaussian priors leads to much lower Rademacher complexity, and is superior to optimally weighted model interpolation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/08/2018

Zero-Resource Multilingual Model Transfer: Learning What to Share

Modern natural language processing and understanding applications have e...
research
01/30/2020

Do We Need Word Order Information for Cross-lingual Sequence Labeling

Most of the recent work in cross-lingual adaptation does not consider th...
research
07/22/2020

Effects of Language Relatedness for Cross-lingual Transfer Learning in Character-Based Language Models

Character-based Neural Network Language Models (NNLM) have the advantage...
research
08/14/2018

Adversarial Neural Networks for Cross-lingual Sequence Tagging

We study cross-lingual sequence tagging with little or no labeled data i...
research
01/23/2023

Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning

Most Transformer language models are primarily pretrained on English tex...

Please sign up or login with your details

Forgot password? Click here to reset