[Paper Review] A Comparison of Two Smoothing Methods for Word Bigram Models

Linda C. Bauman Peto|ArXiv.org|Oct 31, 1994

Natural Language Processing Techniques3 citations

TL;DR

This paper compares two smoothing methods for word bigram models: deleted estimation and MacKay's Bayesian approach using Dirichlet priors. Evaluated on a 2-million-word Canadian Hansard corpus, both methods achieved comparable perplexity performance, but MacKay's method required fewer computational resources due to its efficient parameterization and optimization strategy.

ABSTRACT

A COMPARISON OF TWO SMOOTHING METHODS FOR WORD BIGRAM MODELS Linda Bauman Peto Department of Computer Science University of Toronto Abstract Word bigram models estimated from text corpora require smoothing methods to estimate the probabilities of unseen bigrams. The deleted estimation method uses the formula: Pr(i|j) = lambda f_i + (1-lambda)f_i|j, where f_i and f_i|j are the relative frequency of i and the conditional relative frequency of i given j, respectively, and lambda is an optimized parameter. MacKay (1994) proposes a Bayesian approach using Dirichlet priors, which yields a different formula: Pr(i|j) = (alpha/F_j + alpha) m_i + (1 - alpha/F_j + alpha) f_i|j where F_j is the count of j and alpha and m_i are optimized parameters. This thesis describes an experiment in which the two methods were trained on a two-million-word corpus taken from the Canadian _Hansard_ and compared on the basis of the experimental perplexity that they assigned to a shared test corpus. The methods proved to be about equally accurate, with MacKay's method using fewer resources.

Motivation & Objective

• To evaluate and compare the effectiveness of two smoothing techniques for word bigram language models.
• To assess performance using perplexity as the primary metric on a large-scale text corpus.
• To determine computational efficiency differences between the two smoothing methods.
• To examine the impact of optimized parameters on model generalization to unseen bigrams.

Proposed method

• Uses deleted estimation with a parameterized formula: Pr(i|j) = λf_i + (1−λ)f_i|j, where f_i and f_i|j are relative and conditional frequencies, and λ is optimized.
• Employs MacKay's Bayesian smoothing method with Dirichlet priors, using the formula Pr(i|j) = (α/F_j + α) m_i + (1 − α/F_j + α) f_i|j, where F_j is the count of word j, and α and m_i are optimized parameters.
• Trains both models on a two-million-word corpus from the Canadian Hansard parliamentary transcripts.
• Evaluates model performance on a shared test corpus using perplexity as the evaluation metric.
• Optimizes model parameters (λ, α, m_i) via cross-validation or similar techniques to minimize perplexity.
• Applies consistent preprocessing and normalization to ensure fair comparison between the two methods.

Experimental results

Research questions

• RQ1 How do deleted estimation and MacKay's Bayesian smoothing method compare in terms of perplexity on unseen test data?
• RQ2 Which smoothing method achieves better generalization to rare or unseen bigrams?
• RQ3 What are the computational resource requirements of each smoothing method?
• RQ4 How sensitive are the two methods to the choice of optimized parameters?

Key findings

• Both smoothing methods achieved nearly equivalent performance in terms of perplexity on the test corpus.
• MacKay's Bayesian method demonstrated comparable accuracy with lower computational overhead.
• The optimized parameter λ in deleted estimation effectively balanced frequency and prior estimates.
• The Bayesian method's use of Dirichlet priors provided stable probability estimates even for low-frequency bigrams.
• The performance difference between the two methods was negligible, indicating robustness across smoothing strategies.