DeepAI AI Chat
Log In Sign Up

Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction

by   Paula Czarnowska, et al.
University of Cambridge
Sebastian Ruder

Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language. When translating from Spanish, a good translator would have no problem identifying the proper translation of a statistically rare inflection such as habláramos. Note the lexeme itself, hablar, is relatively common. In this work, we investigate whether state-of-the-art bilingual lexicon inducers are capable of learning this kind of generalization. We introduce 40 morphologically complete dictionaries in 10 languages and evaluate three of the state-of-the-art models on the task of translation of less frequent morphological forms. We demonstrate that the performance of state-of-the-art models drops considerably when evaluated on infrequent morphological inflections and then show that adding a simple morphological constraint at training time improves the performance, proving that the bilingual lexicon inducers can benefit from better encoding of morphology.


page 5

page 7


Better Character Language Modeling Through Morphology

We incorporate morphological supervision into character language models ...

Role of Morphology Injection in Statistical Machine Translation

Phrase-based Statistical models are more commonly used as they perform o...

Cognate-aware morphological segmentation for multilingual neural translation

This article describes the Aalto University entry to the WMT18 News Tran...

Morphology Generation for Statistical Machine Translation

When translating into morphologically rich languages, Statistical MT app...

A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration

One of the primary tasks of morphological parsers is the disambiguation ...

On the Diachronic Stability of Irregularity in Inflectional Morphology

Many languages' inflectional morphological systems are replete with irre...

A Finite State Transducer Based Morphological Analyzer of Maithili Language

Morphological analyzers are the essential milestones for many linguistic...