[Paper Review] Exploiting Diversity in Natural Language Processing: Combining Parsers

John C. Henderson, Eric Brill|ArXiv.org|Jun 1, 2000

Natural Language Processing TechniquesComputer Science7 references106 citations

TL;DR

This paper proposes combining three state-of-the-art statistical parsers using two novel approaches—parser switching and parse hybridization—to improve parsing accuracy on the Penn Treebank. By employing constituent voting and naïve Bayes classification, the method achieves a new SOTA F1 score of 91.25, reducing precision error by 30% and recall error by 6% compared to the best prior result.

ABSTRACT

Three state-of-the-art statistical parsers are combined to produce more accurate parses, as well as new bounds on achievable Treebank parsing accuracy. Two general approaches are presented and two combination techniques are described for each approach. Both parametric and non-parametric models are explored. The resulting parsers surpass the best previously published performance results for the Penn Treebank.

Motivation & Objective

• To improve parsing accuracy on the Penn Treebank by combining multiple independent, high-performing statistical parsers.
• To investigate whether combining parsers with diverse error patterns can reduce overall parsing error beyond individual system performance.
• To evaluate both non-parametric (e.g., constituent voting) and parametric (e.g., naïve Bayes) combination techniques for robustness and accuracy.
• To establish new upper bounds on achievable parsing accuracy using oracle systems and to compare practical combination methods against these bounds.
• To test the robustness of combination techniques when introducing a low-performing parser into the ensemble.

Proposed method

• Uses three high-accuracy statistical parsers—two from the literature and one from the authors—as input for combination.
• Applies constituent voting: a non-parametric method where a constituent is included in the final parse if at least two of the three parsers agree.
• Employs a parametric naïve Bayes classifier to estimate the probability that a constituent should be included, based on individual parser reliability.
• Models the combination as a Bayesian inference problem: the final parse is the set of constituents with P(π(c)=t | M₁(c), ..., Mₖ(c)) > 0.5.
• Estimates probabilities using training data: P(π(c)=t) = N(π(c)=t)/|C| and P(Mᵢ(c)|π(c)=t) = N(Mᵢ(c), π(c)=t)/N(π(c)=t).
• Introduces robustness testing by adding a weak, non-lexicalized PCFG parser to evaluate performance degradation under poor input.

Experimental results

Research questions

• RQ1 Can combining multiple independent, accurate parsers reduce overall parsing error beyond the best individual system?
• RQ2 Do parametric combination methods like naïve Bayes outperform non-parametric methods like constituent voting in terms of accuracy and robustness?
• RQ3 How does the inclusion of a low-performing parser affect the performance of different combination techniques?
• RQ4 What is the upper bound on parsing accuracy achievable with an optimal combination of parsers, as measured by oracle systems?
• RQ5 Are the combination techniques robust to the presence of a poorly performing parser in the ensemble?

Key findings

• The constituent voting and naïve Bayes hybridization techniques both achieved an F1 score of 91.25 on the Penn Treebank test set, surpassing the best previously published result of 89.67.
• The constituent voting method achieved a precision of 92.42% and recall of 90.10%, representing a 30% reduction in precision error and a 6% reduction in recall error compared to the best individual parser.
• The naïve Bayes model outperformed the non-parametric similarity switching technique, with a 90.82% F1 compared to 90.40% for similarity switching, and the difference was statistically significant (α < 0.01).
• The Bayes switching model used the most accurate parser (Parser 3) 71% of the time and the least accurate (Parser 1) only 16% of the time, indicating effective model weighting.
• Robustness testing showed that the parametric methods (Bayes switching and naïve Bayes) maintained high performance even when a weak PCFG parser was introduced, while constituent voting saw a significant drop in precision.
• The maximum precision oracle achieved 100.00% precision and 95.91% recall, setting a new upper bound on achievable performance, and the best combination method reached 97.91% F1, indicating substantial room for improvement.