[Paper Review] Natural Language Parsing as Statistical Pattern Recognition

David M. Magerman|arXiv (Cornell University)|May 3, 1994

Natural Language Processing TechniquesComputer Science50 references226 citations

TL;DR

This paper proposes a statistical pattern recognition approach to natural language parsing, training a parser on annotated corpora without linguistic rules. Using a maximum-entropy model over syntactic features, it achieves 78% accuracy on a test set—surpassing a grammar-based parser's 69%—though only 35% of parses exactly match gold standards, highlighting limitations in feature representation and morphological generalization.

ABSTRACT

Traditional natural language parsers are based on rewrite rule systems developed in an arduous, time-consuming manner by grammarians. A majority of the grammarian's efforts are devoted to the disambiguation process, first hypothesizing rules which dictate constituent categories and relationships among words in ambiguous sentences, and then seeking exceptions and corrections to these rules. In this work, I propose an automatic method for acquiring a statistical parser from a set of parsed sentences which takes advantage of some initial linguistic input, but avoids the pitfalls of the iterative and seemingly endless grammar development process. Based on distributionally-derived and linguistically-based features of language, this parser acquires a set of statistical decision trees which assign a probability distribution on the space of parse trees given the input sentence. These decision trees take advantage of significant amount of contextual information, potentially including all of the lexical information in the sentence, to produce highly accurate statistical models of the disambiguation process. By basing the disambiguation criteria selection on entropy reduction rather than human intuition, this parser development method is able to consider more sentences than a human grammarian can when making individual disambiguation rules. In experiments between a parser, acquired using this statistical framework, and a grammarian's rule-based parser, developed over a ten-year period, both using the same training material and test sentences, the decision tree parser significantly outperformed the grammar-based parser on the accuracy measure which the grammarian was trying to maximize, achieving an accuracy of 78% compared to the grammar-based parser's 69%.

Motivation & Objective

• To demonstrate that statistical models trained on annotated corpora can achieve high parsing accuracy without explicit linguistic rules.
• To challenge the dominance of rule-based grammars in parsing by showing statistical models can outperform them on benchmark data.
• To identify the limitations of current statistical parsers, particularly in handling morphological and syntactic generalization.
• To argue that linguists should contribute by identifying disambiguation criteria rather than writing complex rule systems.
• To explore the feasibility of scaling statistical parsing with limited annotated data, avoiding the need for massive corpora.

Proposed method

• Training a maximum-entropy model on a parsed corpus to learn parsing decisions based on local syntactic and lexical features.
• Representing parsing decisions using a feature space that includes part-of-speech tags, word forms, and structural context (e.g., left/right siblings, span length).
• Employing a hierarchical feature representation with bitstring encodings for non-terminal labels, part-of-speech tags, and syntactic features.
• Using a conditional probability model to predict the most likely parse tree given a sentence, maximizing likelihood over training data.
• Applying iterative smoothing and perplexity-based model selection to refine feature weights and improve generalization.
• Evaluating performance using the crossing-brackets measure, a standard metric for syntactic bracketing accuracy.

Experimental results

Research questions

• RQ1 Can a statistical parser trained solely on annotated corpora outperform a grammar-based parser without any linguistic rule engineering?
• RQ2 What is the impact of linguistic feature representation (e.g., morphology, word class) on parsing accuracy and generalization?
• RQ3 Why does a statistically trained parser fail to match human-annotated parses exactly despite high bracketing accuracy?
• RQ4 To what extent can statistical models generalize across morphological variants (e.g., singular/plural nouns, tensed/untensed verbs) without explicit linguistic features?
• RQ5 How can linguists contribute more effectively to statistical parsing if not by writing rules, but by identifying disambiguation criteria?

Key findings

• The statistical parser achieved 78% accuracy on the crossing-brackets measure, significantly outperforming a grammar-based parser that scored 69%.
• Only 35% of the statistical parser’s parses exactly matched the human-annotated gold standard, indicating substantial structural errors despite high bracketing accuracy.
• Even after excluding part-of-speech tagging errors, only about 50% of parses were exactly correct, underscoring the gap between bracket-level and full-structure accuracy.
• The parser struggled with morphological generalization: singular and plural nouns were not treated as related, and untensed verbs were not linked to their tensed forms.
• Error analysis revealed that the lack of linguistic sophistication in feature representation—especially for morphology and syntactic dependencies—was a key limitation.
• The study concludes that while statistical models can replace rule-based systems in parsing, linguists remain essential not for writing rules, but for identifying the disambiguation criteria that statistical models need to learn.