[Paper Review] An Empirical Study of Smoothing Techniques for Language Modeling

Stanley F. Chen, Joshua Goodman|ArXiv.org|Jun 11, 1996

Natural Language Processing Techniques17 references631 citations

TL;DR

This paper presents a comprehensive empirical evaluation of smoothing techniques for n-gram language models, comparing methods like Katz, Jelinek-Mercer, Church-Gale, and novel approaches such as new-avg-count and new-one-count across varying training data sizes, corpora (Brown and Wall Street Journal), and n-gram orders (bigram and trigram). It finds that performance depends heavily on data size and model order, with Church-Gale outperforming others on large bigram sets and the new methods excelling on trigram models, while optimal parameter tuning significantly improves results.

ABSTRACT

We present an extensive empirical comparison of several smoothing techniques in the domain of language modeling, including those described by Jelinek and Mercer (1980), Katz (1987), and Church and Gale (1991). We investigate for the first time how factors such as training data size, corpus (e.g., Brown versus Wall Street Journal), and n-gram order (bigram versus trigram) affect the relative performance of these methods, which we measure through the cross-entropy of test data. In addition, we introduce two novel smoothing techniques, one a variation of Jelinek-Mercer smoothing and one a very simple linear interpolation technique, both of which outperform existing methods.

Motivation & Objective

• To provide a systematic, large-scale empirical comparison of smoothing techniques in language modeling, addressing gaps in prior work that evaluated only a few methods on single corpora and data sizes.
• To investigate how training data size, corpus type (e.g., Brown vs. Wall Street Journal), and n-gram order (bigram vs. trigram) affect the relative performance of smoothing methods.
• To introduce and evaluate two novel smoothing techniques: a Jelinek-Mercer variant and a simple linear interpolation method, assessing their performance relative to established methods.
• To demonstrate the critical impact of parameter tuning on smoothing performance, showing that sub-optimal settings can significantly degrade results.
• To establish a benchmark for future research by measuring performance via cross-entropy on test data and providing implementation details and parameter sensitivity analysis.

Proposed method

• The study evaluates multiple smoothing techniques, including maximum likelihood, additive smoothing, Jelinek-Mercer, Katz, Church-Gale, and novel methods (new-avg-count and new-one-count), using cross-entropy as the primary performance metric.
• For each method, the probability of a word given its n-1 predecessors is estimated using smoothing rules that redistribute probability mass from observed to unseen n-grams.
• The novel method new-avg-count uses a data-driven average count to estimate unseen n-gram probabilities, while new-one-count applies a fixed count adjustment, both designed to be simple and effective.
• Parameter tuning is performed via automated search for methods with tunable parameters (e.g., δ in Katz smoothing, c_min in new-avg-count), with optimal values extrapolated from smaller data sets due to computational constraints.
• Experiments are conducted across multiple training data sizes (from small to large), on two diverse corpora (Brown and Wall Street Journal), and for both bigram and trigram models.
• Performance is measured using cross-entropy: $ \frac{1}{N_T} \sum_{i=1}^{l_T} -\log_2 P_m(t_i) $, where $ P_m $ is the model from method $ m $, and lower values indicate better performance.

Experimental results

Research questions

• RQ1 How does the performance of smoothing techniques vary with training data size across different corpora and n-gram orders?
• RQ2 Which smoothing method performs best on bigram models trained on small versus large training sets?
• RQ3 How do novel smoothing techniques (new-avg-count and new-one-count) compare to established methods like Katz and Church-Gale in terms of cross-entropy?
• RQ4 To what extent does sub-optimal parameter selection degrade the performance of smoothing methods?
• RQ5 How do the relative performances of interpolation-based methods (interp-held-out vs. interp-del-int) compare, and what explains their differences?

Key findings

• Church-Gale smoothing outperforms all other methods on bigram models trained on large datasets, achieving the lowest cross-entropy.
• Katz smoothing performs best on bigram models trained on smaller datasets and also shows strong performance on trigram models trained on large data.
• The novel method new-avg-count consistently outperforms existing techniques on trigram models across all data sizes and corpora.
• The method new-one-count performs nearly as well as new-avg-count on trigram models and significantly outperforms existing methods, despite being extremely simple to implement.
• Sub-optimal parameter selection can lead to substantial performance degradation, with optimal values being highly dependent on training data size.
• Performance varies significantly with training data size and n-gram order, but remains relatively stable across different corpora, indicating that data size and model complexity are more influential than corpus type.