[Paper Review] Improving Neural Parsing by Disentangling Model Combination and Reranking Effects

Daniel Fried, Mitchell Stern+1 more|arXiv (Cornell University)|Jul 10, 2017

Natural Language Processing Techniques10 references8 citations

TL;DR

This paper proposes a method to disentangle model combination and reranking effects in neural constituency parsing by enabling direct beam search in generative neural parsers. It shows that performance gains from using generative models to rerank base parser outputs are largely due to implicit model combination, not superior generative modeling; explicitly combining scores from base and generative models yields further improvements, achieving new state-of-the-art F1 scores of 94.25 on the Penn Treebank (gold data) and 94.66 with external silver data.

ABSTRACT

Recent work has proposed several generative neural models for constituency parsing that achieve state-of-the-art results. Since direct search in these generative models is difficult, they have primarily been used to rescore candidate outputs from base parsers in which decoding is more straightforward. We first present an algorithm for direct search in these generative models. We then demonstrate that the rescoring results are at least partly due to implicit model combination rather than reranking effects. Finally, we show that explicit model combination can improve performance even further, resulting in new state-of-the-art numbers on the PTB of 94.25 F1 when training only on gold data and 94.66 F1 when using external data.

Motivation & Objective

• To investigate whether performance gains from using generative neural models to rerank outputs of base parsers are due to superior generative modeling or implicit model combination.
• To develop a beam search algorithm that enables direct inference in generative neural parsing models, independent of base parsers.
• To isolate and quantify the contributions of reranking versus model combination in hybrid parsing systems.
• To improve parsing performance through explicit model combination of scores and candidate parses from multiple models.
• To achieve new state-of-the-art results on the Penn Treebank using both gold and external silver-standard training data.

Proposed method

• Proposes an action-synchronous beam search with an augmented state space to enable direct decoding in generative neural parsing models, such as the Recurrent Neural Network Grammar (RG) and LSTM Language Modeling (LM) parsers.
• Uses the same action space (NT, Shift/Gen, Reduce) as the discriminative parser RD, but modifies constraints to allow valid parse generation under the generative model’s joint probability distribution.
• Employs beam search with pruning based on the generative model’s probability, enabling exploration of high-probability parses without relying on a base parser’s output.
• Introduces explicit model combination by averaging scores from the base parser (RD) and generative models (RG, LM) when selecting the best parse from a candidate list.
• Applies candidate augmentation by including parses found via direct search in the generative models to the candidate list of the base parser.
• Uses ensembling of multiple models and interpolation of scores to further improve performance, with hyperparameters tuned on development data.

Experimental results

Research questions

• RQ1 To what extent are performance gains from generative reranking due to reranking effects versus implicit model combination with the base parser?
• RQ2 Can direct search in generative neural parsing models yield better performance than the standard cross-scoring setup (B → A)?
• RQ3 Does explicitly combining scores from a base parser and a generative model improve performance beyond the standard reranking setup?
• RQ4 Are the generative models themselves capable of achieving strong performance independently, or do they rely on the base parser for effective decoding?
• RQ5 Can explicit model combination and candidate augmentation lead to new state-of-the-art results on the Penn Treebank?

Key findings

• Direct beam search in generative models (A → A) yields lower performance than the standard cross-scoring setup (B → A), indicating that the generative model prefers some high-probability parses that are actually detrimental to parsing accuracy.
• The performance gain from B → A is primarily due to implicit model combination rather than superior generative modeling, as the generative model's output is not optimal when used in isolation.
• Explicitly combining scores from the base parser and generative models (e.g., RD + RG or RD + LM) significantly improves performance over using only the generative model’s scores.
• Adding parses found via direct search in the generative models to the candidate list further improves performance, demonstrating the value of candidate augmentation.
• The best result of 94.25 F1 is achieved on the Penn Treebank using only gold data by combining scores and candidates from multiple models, setting a new state-of-the-art.
• With external silver data, the method achieves 94.66 F1, further improving upon prior SOTA, and outperforms both ensembling and standard reranking approaches.