MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

07/30/2020
by   Shayne Longpre, et al.
0

Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). The goal of this dataset is to provide a challenging benchmark for question answering quality across a wide set of languages. Answers are based on a language-independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark state-of-the-art extractive question answering baselines, trained on Natural Questions, including Multilingual BERT, and XLM-RoBERTa, in zero shot and translation settings. Results indicate this dataset is challenging, especially in low-resource languages.

READ FULL TEXT

page 7

page 9

research
07/26/2021

One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval

We present CORA, a Cross-lingual Open-Retrieval Answer Generation model ...
research
10/14/2021

Cross-Lingual GenQA: A Language-Agnostic Generative Question Answering Approach for Open-Domain Question Answering

Open-Retrieval Generative Question Answering (GenQA) is proven to delive...
research
12/18/2021

Cascading Adaptors to Leverage English Data to Improve Performance of Question Answering for Low-Resource Languages

Transformer based architectures have shown notable results on many down ...
research
03/10/2020

TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

Confidently making progress on multilingual modeling requires challengin...
research
05/20/2022

Down and Across: Introducing Crossword-Solving as a New NLP Benchmark

Solving crossword puzzles requires diverse reasoning capabilities, acces...
research
04/16/2019

Query Expansion for Cross-Language Question Re-Ranking

Community question-answering (CQA) platforms have become very popular fo...
research
01/02/2022

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Autograding short textual answers has become much more feasible due to t...

Please sign up or login with your details

Forgot password? Click here to reset