DeepAI AI Chat
Log In Sign Up

Open-Set Language Identification

by   Shervin Malmasi, et al.
Harvard University

We present the first open-set language identification experiments using one-class classification. We first highlight the shortcomings of traditional feature extraction methods and propose a hashing-based feature vectorization approach as a solution. Using a dataset of 10 languages from different writing systems, we train a One- Class Support Vector Machine using only a monolingual corpus for each language. Each model is evaluated against a test set of data from all 10 languages and we achieve an average F-score of 0.99, highlighting the effectiveness of this approach for open-set language identification.


page 1

page 2

page 3

page 4


Automatic Language Identification for Celtic Texts

Language identification is an important Natural Language Processing task...

Modernizing Open-Set Speech Language Identification

While most modern speech Language Identification methods are closed-set,...

Native Language Identification using Stacked Generalization

Ensemble methods using multiple classifiers have proven to be the most s...

AfroLID: A Neural Language Identification Tool for African Languages

Language identification (LID) is a crucial precursor for NLP, especially...

Tuplemax Loss for Language Identification

In many scenarios of a language identification task, the user will speci...

Language Model Adaptation for Language and Dialect Identification of Text

This article describes an unsupervised language model adaptation approac...

A Semisupervised Approach for Language Identification based on Ladder Networks

In this study we address the problem of training a neuralnetwork for lan...