Consistency Regularization for Cross-Lingual Fine-Tuning

06/15/2021
by   Bo Zheng, et al.
0

Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/29/2020

Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning

Recently, fine-tuning pre-trained cross-lingual models (e.g., multilingu...
research
08/31/2021

Cross-Lingual Text Classification of Transliterated Hindi and Malayalam

Transliteration is very common on social media, but transliterated text ...
research
06/08/2023

T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification

Cross-lingual text classification leverages text classifiers trained in ...
research
01/13/2023

Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing

Standard fine-tuning of language models typically performs well on in-di...
research
07/25/2023

XDLM: Cross-lingual Diffusion Language Model for Machine Translation

Recently, diffusion models have excelled in image generation tasks and h...
research
10/14/2021

Composable Sparse Fine-Tuning for Cross-Lingual Transfer

Fine-tuning all parameters of a pre-trained model has become the mainstr...

Please sign up or login with your details

Forgot password? Click here to reset