[Paper Review] Compositional Morphology for Word Representations and Language Modelling

Jan A. Botha, Phil Blunsom|arXiv (Cornell University)|May 16, 2014

Topic Modeling45 references150 citations

TL;DR

This paper proposes a compositional morphology-aware continuous-space language model (CSLM) that represents words as additive combinations of morpheme vectors, improving performance on morphologically rich languages. By integrating these morphologically informed representations into a log-bilinear language model, the approach reduces perplexity and boosts translation performance by up to 1.2 BLEU points on morphologically complex languages like Russian and Czech.

ABSTRACT

This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model. Our approach is evaluated in the context of log-bilinear language models, rendered suitably efficient for implementation inside a machine translation decoder by factoring the vocabulary. We perform both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that our model learns morphological representations that both perform well on word similarity tasks and lead to substantial reductions in perplexity. When used for translation into morphologically rich languages with large vocabularies, our models obtain improvements of up to 1.2 BLEU points relative to a baseline system using back-off n-gram models.

Motivation & Objective

• Address data sparsity in statistical language models caused by morphological variation in rich languages.
• Integrate morphological structure into probabilistic continuous-space language models (CSLMs) without relying on handcrafted features.
• Enable effective out-of-vocabulary (OOV) word handling through compositional vector representations.
• Improve both intrinsic (word similarity) and extrinsic (machine translation) performance in low-resource and morphologically complex settings.
• Demonstrate scalability and integration into real-world machine translation decoders using efficient factorization and classing techniques.

Proposed method

• Represent words as the sum of factor vectors, where each factor corresponds to a morpheme (e.g., stem, affix) or surface form.
• Use a deterministic morphological segmentation function μ to map each word to a variable-length sequence of factors.
• Compute word representations as additive combinations: r̃_v = Σ_{f∈μ(v)} r_f, enabling shared statistical strength across related forms.
• Incorporate the full surface form as a factor to preserve non-compositional word forms and break order invariance.
• Apply word classing to speed up inference, enabling integration into a decoder for end-to-end machine translation.
• Train the model using a log-bilinear language model (LBL) framework with normalized probability estimation for efficient decoding.

Experimental results

Research questions

• RQ1 Can compositional morphological representations improve intrinsic word similarity performance in multiple languages?
• RQ2 Does incorporating morphological structure into a continuous-space language model reduce perplexity compared to baseline n-gram and CSLM models?
• RQ3 To what extent do morphology-aware language models improve machine translation quality, especially for morphologically rich languages?
• RQ4 How effective are the learned morpheme vectors in handling out-of-vocabulary (OOV) words in low-resource settings?
• RQ5 Can the proposed model be efficiently integrated into a real-time machine translation decoder without sacrificing performance?

Key findings

• The proposed CLBL++ model achieved up to 1.2 BLEU point improvement over baseline systems in English-to-Russian translation, with 1.0 point gains in English-to-Czech and English-to-Spanish.
• The model reduced perplexity significantly across all tested languages, with the most substantial gains observed in morphologically rich languages like Russian and Czech.
• Word similarity correlations with human ratings improved across multiple languages, outperforming a more complex model that used larger training data.
• The additive composition of morpheme vectors enabled effective OOV word representation, particularly benefiting low-frequency and unseen word forms.
• The integration of the normalized CSLM into the cdec decoder was feasible and efficient due to word classing, enabling real-time decoding.
• The model’s gains were most pronounced in morphologically complex languages, while improvements in German were limited by bilingual coverage rather than language model quality.