[Paper Review] Inducing Probabilistic Grammars by Bayesian Model Merging

Andreas Stolcke, Stephen M. Omohundro|arXiv (Cornell University)|Sep 13, 1994

Natural Language Processing Techniques16 references17 citations

TL;DR

This paper proposes a Bayesian model merging framework for inducing probabilistic grammars from positive corpora by iteratively generalizing an initial, data-specific model through merging operations. By maximizing the posterior probability—balancing data fit and model simplicity—it achieves compact, generalizable HMMs, class-based n-grams, and stochastic context-free grammars, outperforming traditional methods like Baum-Welch in structure induction.

ABSTRACT

We describe a framework for inducing probabilistic grammars from corpora of positive samples. First, samples are {\em incorporated} by adding ad-hoc rules to a working grammar; subsequently, elements of the model (such as states or nonterminals) are {\em merged} to achieve generalization and a more compact representation. The choice of what to merge and when to stop is governed by the Bayesian posterior probability of the grammar given the data, which formalizes a trade-off between a close fit to the data and a default preference for simpler models (`Occam's Razor'). The general scheme is illustrated using three types of probabilistic grammars: Hidden Markov models, class-based $n$-grams, and stochastic context-free grammars.

Motivation & Objective

• To address the challenge of discovering optimal discrete structures for probabilistic models from training data, a key bottleneck in applications like speech recognition and NLP.
• To formalize model generalization as a Bayesian inference problem, trading off data fit against model simplicity using Occam’s Razor.
• To develop a unified, principled framework for probabilistic grammar induction applicable across HMMs, n-grams, and SCFGs.
• To demonstrate that merging based on posterior probability outperforms standard likelihood maximization in structure induction.

Proposed method

• Start with an initial model M₀ that explicitly encodes each training sample as a unique path, maximizing likelihood but lacking generalization.
• Apply iterative merging operations that coalesce substructures (e.g., states or nonterminals), sharing transitions and emissions from merged components.
• Use the Bayesian posterior P(M|X) = P(M)P(X|M)/P(X) as the criterion to guide merging, favoring models that balance fit and simplicity.
• Employ best-first or beam search to explore merging steps that maximize the immediate posterior increase, stopping when no further gain is possible.
• Use on-line interleaving of data incorporation and merging to maintain manageable model size.
• Define priors P(M) to encode structural preferences, with the posterior naturally penalizing overgeneralization based on absolute data frequency.

Experimental results

Research questions

• RQ1 Can a Bayesian posterior criterion effectively guide the generalization of probabilistic grammar structures from positive training samples?
• RQ2 How does model merging based on posterior probability compare to traditional likelihood maximization (e.g., Baum-Welch) in discovering correct HMM topologies?
• RQ3 To what extent does the Bayesian framework prevent overgeneralization when data frequency is high but the underlying structure is complex?
• RQ4 Can the same merging framework be uniformly applied across diverse probabilistic models like HMMs, n-grams, and SCFGs?
• RQ5 How does the inclusion of absolute data frequency in the likelihood term affect generalization compared to relative frequency-only approaches?

Key findings

• The Bayesian model merging approach successfully induces compact, generalizable HMMs that generalize beyond the finite training set, such as capturing the infinite language {ab}n from finite samples.
• For HMMs, the method achieves better structure induction than Baum-Welch estimation, particularly in recovering the true underlying topology.
• The SCFG version of the algorithm simplifies and unifies prior approaches to CFG induction, providing a principled way to balance data fit and model complexity.
• The framework penalizes overgeneralization based on absolute data frequency: a 100-fold increase in sample frequency of a pattern leads to a 100-fold increase in log-likelihood loss for generalization, making overgeneralization less likely.
• The posterior-based merging process naturally stops at optimal models, avoiding local maxima when combined with lookahead in search.
• The method is robust and cognitively plausible, with merging operations conceptually resembling equivalence-class construction in non-probabilistic grammar induction, suggesting broader applicability.