DeepAI AI Chat
Log In Sign Up

Multilingual Open Text 1.0: Public Domain News in 44 Languages

by   Chester Palen-Michel, et al.
Brandeis University

We present a new multilingual corpus containing text in 44 languages, many of which have relatively few existing resources for natural language processing. The first release of the corpus contains over 2.7 million news articles and 1 million shorter passages published between 2001–2021, collected from Voice of America news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published.


page 1

page 2

page 3

page 4


Bianet: A Parallel News Corpus in Turkish, Kurdish and English

We present a new open-source parallel corpus consisting of news articles...

LR-Sum: Summarization for Less-Resourced Languages

This preprint describes work in progress on LR-Sum, a new permissively-l...

PerKey: A Persian News Corpus for Keyphrase Extraction and Generation

Keyphrases provide an extremely dense summary of a text. Such informatio...

Identifying civilians killed by police with distantly supervised entity-event extraction

We propose a new, socially-impactful task for natural language processin...

Upgrading the Newsroom: An Automated Image Selection System for News Articles

We propose an automated image selection system to assist photo editors i...

Creating a contemporary corpus of similes in Serbian by using natural language processing

Simile is a figure of speech that compares two things through the use of...

Linguistic Cues of Deception in a Multilingual April Fools' Day Context

In this work we consider the collection of deceptive April Fools' Day(AF...