[Paper Review] Aggregate and mixed-order Markov models for statistical language processing

Lawrence K. Saul, Fernando Pereira|ArXiv.org|Jun 9, 1997

Algorithms and Data CompressionComputer Science12 references138 citations

TL;DR

This paper proposes aggregate and mixed-order Markov models as intermediate language models between standard n-gram models, using EM-trained probabilistic word classes and skip-k transition matrices to improve generalization. These models reduce perplexity on unseen word combinations by over 50% when interposed in smoothing procedures, outperforming traditional n-gram backoff models with significantly less training time.

ABSTRACT

We consider the use of language models whose size and accuracy are intermediate between different order n-gram models. Two types of models are studied in particular. Aggregate Markov models are class-based bigram models in which the mapping from words to classes is probabilistic. Mixed-order Markov models combine bigram models whose predictions are conditioned on different words. Both types of models are trained by Expectation-Maximization (EM) algorithms for maximum likelihood estimation. We examine smoothing procedures in which these models are interposed between different order n-grams. This is found to significantly reduce the perplexity of unseen word combinations.

Motivation & Objective

• To address the challenge of generalizing from sparse training data in large-vocabulary language modeling.
• To develop models that are intermediate in size and accuracy between different n-gram orders.
• To reduce perplexity on unseen word combinations through improved smoothing techniques.
• To provide a computationally efficient alternative to maximum-entropy models with comparable or better performance.

Proposed method

• Aggregate Markov models use probabilistic word classes to reduce parameter count, with class assignments learned via EM for maximum likelihood.
• Mixed-order Markov models combine predictions from skip-k transition matrices (e.g., conditioned on word k steps back) using context-dependent mixing coefficients.
• Both models are trained using Expectation-Maximization (EM) algorithms to jointly optimize mixing weights and transition probabilities.
• Smoothing is achieved by interposing these intermediate models between different n-gram orders, improving generalization to rare or unseen sequences.
• The models scale as O(mV²) for mixed-order models, significantly less than full m+1-gram models at O(V^{m+1}).
• The framework allows for consistent interpolation by adjusting predictions to match the contexts in which they are used.

Experimental results

Research questions

• RQ1 Can models that are intermediate in complexity between unigram, bigram, and trigram n-grams reduce perplexity on unseen word sequences?
• RQ2 How effective is the use of probabilistic word classes (aggregate models) in improving generalization for large-vocabulary language modeling?
• RQ3 Can skip-k transition matrices, conditioned on non-adjacent words, enhance prediction accuracy beyond standard bigram models?
• RQ4 Does inserting intermediate models between n-gram orders in smoothing procedures yield significant improvements in perplexity?
• RQ5 How does the training efficiency of EM-based intermediate models compare to maximum-entropy models?

Key findings

• The use of intermediate aggregate and mixed-order models reduced perplexity on unseen word combinations by over 50% compared to baseline trigram backoff models.
• Mixed-order models achieved a perplexity reduction of 16% to 22%, depending on trigram truncation, outperforming baseline trigram models.
• The EM-based training of mixed-order models completed in less than 12 CPU-hours, significantly faster than the 200 CPU-days required by maximum-entropy models in comparable work.
• Truncating rare trigrams improved performance, with the mixed-order model maintaining low perplexity even when only the most frequent trigrams were retained.
• The models demonstrated consistent performance across different random test set splits, indicating robustness.
• Aggregate models achieved a trade-off between unigram and bigram complexity by controlling the number of word classes, with performance improving as class count increased.