Hierarchical Character-Word Models for Language Identification

08/10/2016

∙

by Aaron Jaech, et al.

∙

University of Washington

∙

Social media messages' brevity and unconventional spelling pose a challenge to language identification. We introduce a hierarchical model that learns character and contextualized word-level representations for language identification. Our method performs well against strong base- lines, and can also reveal code-switching.

READ FULL TEXT

Hierarchical Character-Word Models for Language Identification

Language Identification of Bengali-English Code-Mixed data using Character & Phonetic based LSTM Models

Ghmerti at SemEval-2019 Task 6: A Deep Word- and Character-based Approach to Offensive Language Identification

Sentence-level dialects identification in the greater China region

Tweet2Vec: Character-Based Distributed Representations for Social Media

A Sub-Character Architecture for Korean Language Processing

All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media

Semantic Classification of Tabular Datasets via Character-Level Convolutional Neural Networks

Code Repositories

twitter_langid

Hierarchical Character-Word Models for Language Identification

Related Research

Language Identification of Bengali-English Code-Mixed data using Character & Phonetic based LSTM Models

Ghmerti at SemEval-2019 Task 6: A Deep Word- and Character-based Approach to Offensive Language Identification

Sentence-level dialects identification in the greater China region

Tweet2Vec: Character-Based Distributed Representations for Social Media

A Sub-Character Architecture for Korean Language Processing

All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media

Semantic Classification of Tabular Datasets via Character-Level Convolutional Neural Networks

Code Repositories

twitter_langid