[Paper Review] Entropy-based Pruning of Backoff Language Models

Andreas Stolcke|ArXiv.org|Jun 11, 2000

Natural Language Processing Techniques9 references267 citations

TL;DR

This paper proposes an entropy-based pruning method for N-gram backoff language models that minimizes relative entropy between the original and pruned models to preserve performance while reducing model size. The method computes exact relative entropy changes efficiently, enabling a threshold-based pruning strategy that reduces a Hub4 4-gram model to 26% of its original size with no significant increase in recognition error.

ABSTRACT

A criterion for pruning parameters from N-gram backoff language models is developed, based on the relative entropy between the original and the pruned model. It is shown that the relative entropy resulting from pruning a single N-gram can be computed exactly and efficiently for backoff models. The relative entropy measure can be expressed as a relative change in training set perplexity. This leads to a simple pruning criterion whereby all N-grams that change perplexity by less than a threshold are removed from the model. Experiments show that a production-quality Hub4 LM can be reduced to 26% its original size without increasing recognition error. We also compare the approach to a heuristic pruning criterion by Seymore and Rosenfeld (1996), and show that their approach can be interpreted as an approximation to the relative entropy criterion. Experimentally, both approaches select similar sets of N-grams (about 85% overlap), with the exact relative entropy criterion giving marginally better performance.

Motivation & Objective

• To develop a principled, information-theoretic criterion for pruning N-gram backoff language models that optimizes model quality and size.
• To ensure the pruning criterion is computationally efficient and self-contained, relying only on model parameters without external training statistics.
• To evaluate the proposed relative entropy criterion against existing heuristic methods, particularly Seymore and Rosenfeld’s approach.
• To demonstrate that pruning based on relative entropy maintains or slightly improves recognition performance while drastically reducing model size.

Proposed method

• The method uses relative entropy (Kullback-Leibler divergence) as a measure of model distortion between the original and pruned language models.
• It computes the exact relative entropy contribution of pruning each individual N-gram, enabling precise ranking of N-grams by their impact on model quality.
• The pruning threshold is set based on the relative change in perplexity, with N-grams causing a perplexity increase below a threshold being removed.
• After pruning, backoff weights are recomputed to maintain probability normalization in the pruned model.
• The method is self-contained, requiring only model parameters (probabilities and backoff weights), not raw training counts.
• The approach is compared to Seymore and Rosenfeld’s heuristic criterion, which uses a weighted difference of log probabilities and frequency, shown to be an approximation of the relative entropy criterion.

Experimental results

Research questions

• RQ1 Can relative entropy between the original and pruned language model serve as a sound, efficient, and self-contained criterion for N-gram pruning?
• RQ2 How does the performance of entropy-based pruning compare to the heuristic method proposed by Seymore and Rosenfeld in terms of perplexity and word error rate?
• RQ3 To what extent do the N-grams selected by the relative entropy and heuristic criteria overlap, and how does this affect model quality?
• RQ4 Can the relative entropy criterion be computed efficiently enough to be practical for large-scale language models such as the Hub4 model?
• RQ5 Does pruning based on relative entropy maintain recognition performance while significantly reducing model size?

Key findings

• The entropy-based pruning method reduced a production-quality Hub4 4-gram language model to 26% of its original size with no significant increase in recognition error.
• The relative entropy criterion achieved a 1.2% lower perplexity than the Seymore and Rosenfeld heuristic method at 100,000 trigrams, though the difference was negligible in word error rate.
• The two pruning methods selected N-grams with 85% to 88% overlap, indicating strong agreement in N-gram selection despite different underlying criteria.
• The relative entropy method outperformed the heuristic method marginally in perplexity, confirming its theoretical soundness as an optimization criterion.
• The pruned model with 100,000 trigrams achieved a word error rate of 33.1%, matching the full model’s performance, demonstrating that pruning does not degrade recognition quality.
• The method is computationally efficient and self-contained, relying only on model parameters, making it suitable for post-processing existing language models.