From English To Foreign Languages: Transferring Pre-trained Language Models

02/18/2020
by   Ke Tran, et al.
0

Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT base model within a day and a foreign BERT large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.

READ FULL TEXT
research
01/17/2020

RobBERT: a Dutch RoBERTa-based Language Model

Pre-trained language models have been dominating the field of natural la...
research
10/03/2022

Characterization of effects of transfer learning across domains and languages

With ever-expanding datasets of domains, tasks and languages, transfer l...
research
04/15/2022

mGPT: Few-Shot Learners Go Multilingual

Recent studies report that autoregressive language models can successful...
research
04/12/2023

ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

Over the last few years, large language models (LLMs) have emerged as th...
research
01/04/2021

Transformers and Transfer Learning for Improving Portuguese Semantic Role Labeling

Semantic Role Labeling (SRL) is a core Natural Language Processing task....
research
04/07/2022

BERTuit: Understanding Spanish language in Twitter through a native transformer

The appearance of complex attention-based language models such as BERT, ...
research
05/19/2022

Overcoming Language Disparity in Online Content Classification with Multimodal Learning

Advances in Natural Language Processing (NLP) have revolutionized the wa...

Please sign up or login with your details

Forgot password? Click here to reset