What makes multilingual BERT multilingual?

10/20/2020
by   Chi-Liang Liu, et al.
12

Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

04/20/2020

A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT

Recently, multilingual BERT works remarkably well on cross-lingual trans...
08/31/2019

Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrate...
07/01/2021

Multilingual Central Repository: a Cross-lingual Framework for Developing Wordnets

Language resources are necessary for language processing,but building th...
03/16/2022

Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure

Multilingual pre-trained language models, such as mBERT and XLM-R, have ...
08/04/2020

NLPDove at SemEval-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer

This paper describes our approach to the task of identifying offensive l...
03/11/2021

Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings

In this work we look into adding a new language to a multilingual NMT sy...
11/03/2021

Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval

Interactive and non-interactive model are the two de-facto standard fram...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.