PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

by   Yinfei Yang, et al.

Most existing work on adversarial data generation focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. We remedy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacity to capture non-local context and sentence structure, and using different multilingual training and evaluation regimes. Multilingual BERT fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an average accuracy gain of 23 shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenge to drive multilingual research that better captures structure and contextual information.


page 1

page 2

page 3

page 4


PAWS: Paraphrase Adversaries from Word Scrambling

Existing paraphrase identification datasets lack sentence pairs that hav...

Revisiting non-English Text Simplification: A Unified Multilingual Benchmark

Recent advancements in high-quality, large-scale English resources have ...

Slovene SuperGLUE Benchmark: Translation and Evaluation

We present a Slovene combined machine-human translated SuperGLUE benchma...

mMARCO: A Multilingual Version of MS MARCO Passage Ranking Dataset

The MS MARCO ranking dataset has been widely used for training deep lear...

Contextual Label Projection for Cross-Lingual Structure Extraction

Translating training data into target languages has proven beneficial fo...

Large-scale Bilingual Language-Image Contrastive Learning

This paper is a technical report to share our experience and findings bu...

Please sign up or login with your details

Forgot password? Click here to reset