[Paper Review] The Generalized Smallest Grammar Problem

Payam Siyari, Matthias Gallé|arXiv (Cornell University)|Aug 31, 2016

Natural Language Processing Techniques6 references3 citations

TL;DR

This paper proposes a generalized smallest grammar problem that extends traditional straight-line grammars to non-recursive grammars, enabling structural generalization in grammatical inference. By combining Minimum Description Length (MDL) with Harris' substitutability principle, the method infers higher-precision syntactic rules that capture natural language structure more effectively than prior SGP-based approaches, achieving smaller grammars and significantly improved bracketing accuracy on Penn Treebank benchmarks.

ABSTRACT

The Smallest Grammar Problem -- the problem of finding the smallest context-free grammar that generates exactly one given sequence -- has never been successfully applied to grammatical inference. We investigate the reasons and propose an extended formulation that seeks to minimize non-recursive grammars, instead of straight-line programs. In addition, we provide very efficient algorithms that approximate the minimization problem of this class of grammars. Our empirical evaluation shows that we are able to find smaller models than the current best approximations to the Smallest Grammar Problem on standard benchmarks, and that the inferred rules capture much better the syntactic structure of natural language.

Motivation & Objective

• To address the failure of the Smallest Grammar Problem (SGP) in grammatical inference due to its restriction to straight-line programs with no generalization capacity.
• To develop a formalism that enables non-recursive grammars to capture syntactic structure in natural language sequences.
• To design an efficient algorithm that minimizes grammar size while optimizing for structural generalization using MDL and distributional similarity.
• To empirically evaluate the method on standard benchmarks and compare its performance in grammar size and syntactic rule quality against existing SGP algorithms.
• To demonstrate that inferred rules better reflect true syntactic constituents than those from traditional SGP approaches.

Proposed method

• Extends the SGP formalism from straight-line programs to non-recursive grammars to allow structural generalization beyond simple substring replacement.
• Applies the Minimum Description Length (MDL) principle to balance model size and data encoding cost, ensuring efficient and meaningful grammar compression.
• Uses Harris' substitutability theory to identify contextually substitutable substrings as candidates for new non-terminal rules.
• Employs a post-processing step on an initial straight-line grammar to infer branching rules based on context and inside bracket patterns.
• Implements a greedy search strategy to explore the grammar space efficiently, stopping early to control grammar size and improve rule quality.
• Encodes the final grammar using a compact representation that ensures unambiguous reconstruction of the original sequence.

Experimental results

Research questions

• RQ1 Why have previous SGP-based approaches failed in grammatical inference despite theoretical appeal?
• RQ2 Can extending the SGP formalism to non-recursive grammars improve structural generalization in natural language sequences?
• RQ3 To what extent can MDL-based optimization and substitutability principles yield smaller, more accurate grammars than current SGP algorithms?
• RQ4 How does the precision and consistency of inferred syntactic brackets compare between non-recursive and straight-line grammars?
• RQ5 Does the proposed method generalize to longer sequences and complex syntactic structures beyond short sentences?

Key findings

• The proposed method achieves smaller grammars than the current state-of-the-art on standard benchmarks, with a notable improvement on the 'kennedy.xls' sequence.
• The post-processed non-recursive grammars achieve a bracketing precision of 50.48% on 792 additional brackets, significantly higher than the 21% precision of the base greedy algorithm.
• Context brackets are more accurate (up to 60%) but less consistent (85% non-crossing), while inside brackets are more consistent (90%) but less accurate.
• The method maintains high performance on long sequences (over 1.3 million tokens), with no degradation in efficiency or accuracy.
• Straight-line grammars alone achieve only 22% bracketing precision, highlighting the advantage of the proposed non-recursive extension.
• Even on short sentences (≤10 words), context brackets achieve 80% precision, outperforming other unsupervised parsing methods despite low recall.