[Paper Review] Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction

Paula Czarnowska, Sebastian Ruder+3 more|arXiv (Cornell University)|Sep 6, 2019

Natural Language Processing Techniques18 references4 citations

TL;DR

This paper introduces 40 morphologically complete bilingual dictionaries for 10 languages to evaluate morphological generalization in state-of-the-art bilingual lexicon induction (BLI) models. It demonstrates that existing models severely underperform on rare inflections, even of common lexemes, and shows that enforcing a simple morphological constraint at training time significantly improves performance across both related and unrelated language pairs, proving that better morphology encoding enhances generalization.

ABSTRACT

Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language. When translating from Spanish, a good translator would have no problem identifying the proper translation of a statistically rare inflection such as habláramos. Note the lexeme itself, hablar, is relatively common. In this work, we investigate whether state-of-the-art bilingual lexicon inducers are capable of learning this kind of generalization. We introduce 40 morphologically complete dictionaries in 10 languages and evaluate three of the state-of-the-art models on the task of translation of less frequent morphological forms. We demonstrate that the performance of state-of-the-art models drops considerably when evaluated on infrequent morphological inflections and then show that adding a simple morphological constraint at training time improves the performance, proving that the bilingual lexicon inducers can benefit from better encoding of morphology.

Motivation & Objective

• To investigate whether state-of-the-art bilingual lexicon inducers can generalize to rare morphological inflections, as human translators do.
• To address the lack of comprehensive evaluation resources that include full inflectional paradigms, especially for rare forms.
• To propose and evaluate a novel experimental paradigm that independently controls for word frequency, morphology, lexeme frequency, and lexeme identity.
• To test whether enforcing morphological constraints during training improves generalization performance on rare and novel morphological forms.

Proposed method

• Constructed 40 morphologically complete dictionaries for 5 Slavic and 5 Romance languages, covering full inflectional paradigms of all included lexemes.
• Designed a controlled experimental framework to evaluate BLI models under independent variation of four variables: word form frequency, morphosyntactic category, lexeme frequency, and lexeme identity.
• Evaluated three leading BLI models—Artetxe et al. (2017), Ruder et al. (2018), and others—on all 40 language pairs using the new dictionaries.
• Introduced a hard morphological constraint that restricts word alignment during training to only those pairs sharing the same morphosyntactic category (e.g., 3rd person plural past tense).
• Trained models with the morphological constraint using only UniMorph-tagged forms, ensuring alignment only between semantically and grammatically compatible forms.
• Compared performance on standard BLI and controlled-for-lexeme BLI tasks, measuring accuracy across morphosyntactic categories and frequency levels.

Experimental results

Research questions

• RQ1 Can state-of-the-art bilingual lexicon induction models generalize to rare morphological inflections of common lexemes, as human translators do?
• RQ2 How does model performance vary across different morphosyntactic categories and word frequencies in a controlled evaluation setting?
• RQ3 Does enforcing a simple morphological constraint during training improve generalization performance on rare and novel inflections?
• RQ4 How does performance on morphological generalization differ between genetically related and unrelated language pairs?
• RQ5 To what extent do existing evaluation benchmarks fail to capture the true generalization capability of BLI models due to their bias toward high-frequency forms?

Key findings

• Performance of state-of-the-art BLI models drops significantly on rare morphological inflections, even for common lexemes, indicating poor generalization to the long tail of linguistic forms.
• Models generalize well for frequent morphosyntactic categories (e.g., 3rd person singular present) but fail on less frequent categories, especially those with large paradigms.
• The morphological constraint, which restricts alignment to only same-category forms during training, leads to consistent performance improvements across all Romance language pairs and many Slavic pairs.
• On the Polish–Spanish pair (unrelated languages), performance was low overall (28% accuracy for in-vocabulary pairs), but the morphological constraint still improved results, showing its robustness.
• The MUSE benchmark, which focuses on high-frequency forms, is skewed toward frequent words and underrepresents rare inflections, limiting its ability to assess true generalization.
• The study reveals a nuanced failure pattern: models struggle most with rare morphosyntactic categories and large paradigms, even when the lexeme is common.