DeepAI AI Chat
Log In Sign Up

Zero-shot Cross-lingual Transfer is Under-specified Optimization

07/12/2022
by   Shijie Wu, et al.
Johns Hopkins University
0

Pretrained multilingual encoders enable zero-shot cross-lingual transfer, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that this high variance results from zero-shot cross-lingual transfer solving an under-specified optimization problem. We show that any linear-interpolated model between the source language monolingual model and source + target bilingual model has equally low source language generalization error, yet the target language generalization error reduces smoothly and linearly as we move from the monolingual to bilingual model, suggesting that the model struggles to identify good solutions for both source and target languages using the source language alone. Additionally, we show that zero-shot solution lies in non-flat region of target language error generalization surface, causing the high variance.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/04/2023

DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer

Zero-shot cross-lingual transfer is promising, however has been shown to...
09/27/2021

Rumour Detection via Zero-shot Cross-lingual Transfer Learning

Most rumour detection models for social media are designed for one speci...
04/19/2019

Zero-Shot Cross-Lingual Opinion Target Extraction

Aspect-based sentiment analysis involves the recognition of so called op...
12/31/2020

A Closer Look at Few-Shot Crosslingual Transfer: Variance, Benchmarks and Baselines

We present a focused study of few-shot crosslingual transfer, a recently...
10/05/2021

Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance

Multilingual language models achieve impressive zero-shot accuracies in ...
01/24/2023

Cross-lingual German Biomedical Information Extraction: from Zero-shot to Human-in-the-Loop

This paper presents our project proposal for extracting biomedical infor...
02/15/2022

Delving Deeper into Cross-lingual Visual Question Answering

Visual question answering (VQA) is one of the crucial vision-and-languag...