[Paper Review] Cross-Domain Generalization of Neural Constituency Parsers

Daniel Fried, Nikita Kitaev+1 more|arXiv (Cornell University)|Jul 9, 2019

Natural Language Processing Techniques33 references4 citations

TL;DR

This paper investigates zero-shot cross-domain generalization of neural constituency parsers, showing that while pre-trained encoders improve performance across domains, they do not disproportionately help out-of-domain sets. Crucially, structured prediction (via In-Order decoding) significantly enhances exact match accuracy and generalization to longer spans and out-of-domain corpora, even when using strong BERT-based encoders.

ABSTRACT

Neural parsers obtain state-of-the-art results on benchmark treebanks for constituency parsing -- but to what degree do they generalize to other domains? We present three results about the generalization of neural parsers in a zero-shot setting: training on trees from one corpus and evaluating on out-of-domain corpora. First, neural and non-neural parsers generalize comparably to new domains. Second, incorporating pre-trained encoder representations into neural parsers substantially improves their performance across all domains, but does not give a larger relative improvement for out-of-domain treebanks. Finally, despite the rich input representations they learn, neural parsers still benefit from structured output prediction of output trees, yielding higher exact match accuracy and stronger generalization both to larger text spans and to out-of-domain corpora. We analyze generalization on English and Chinese corpora, and in the process obtain state-of-the-art parsing results for the Brown, Genia, and English Web treebanks.

Motivation & Objective

• To evaluate how well neural constituency parsers generalize to out-of-domain text without fine-tuning.
• To compare the zero-shot generalization performance of neural and non-neural parsers across diverse domains.
• To assess whether pre-trained language representations (e.g., BERT) yield greater relative gains on out-of-domain corpora.
• To investigate whether structured output prediction improves generalization beyond strong input representations.
• To establish state-of-the-art results on Brown, Genia, and English Web Treebanks via improved generalization.

Proposed method

• Trained state-of-the-art neural parsers (Chart and In-Order) on the in-domain WSJ and CTB treebanks.
• Evaluated generalization on out-of-domain English (Brown, Genia, EWT) and Chinese (CTB) corpora in a zero-shot setting.
• Integrated BERT-based pre-trained encoders into both neural parser architectures to assess their impact on domain generalization.
• Compared structured decoding (In-Order, which conditions on previously predicted structure) with unstructured decoding (Chart, which predicts spans independently).
• Measured performance using F1 scores and exact match accuracy, with analysis stratified by span length.
• Used standard benchmark splits and reported results on test sets from prior work for direct comparison.

Experimental results

Research questions

• RQ1 Do neural constituency parsers generalize as well as non-neural parsers to out-of-domain corpora in a zero-shot setting?
• RQ2 Does pre-training language representations (e.g., BERT) yield larger relative improvements on out-of-domain treebanks compared to in-domain ones?
• RQ3 Does structured prediction (e.g., via transition-based decoding) improve generalization to longer spans and out-of-domain text, even with strong pre-trained encoders?
• RQ4 Can improvements on in-domain benchmarks (e.g., WSJ) reliably transfer to out-of-domain settings?
• RQ5 How do exact match accuracies compare between structured and unstructured neural parsers across domains and span lengths?

Key findings

• Neural and non-neural parsers generalize comparably to out-of-domain corpora, with similar relative performance degradation across domains.
• Pre-trained BERT encoders improve F1 across all domains, but the relative error reduction is consistent across in-domain and out-of-domain sets, indicating no disproportionate benefit for out-of-domain generalization.
• The In-Order parser (structured decoder) achieves higher exact match accuracy than the Chart parser (unstructured decoder) across all domains, with absolute improvements ranging from 0.5 to 2.8 percentage points.
• For longer spans (≥30–40 words), the In-Order parser significantly outperforms the Chart parser in F1, with the performance gap emerging only at longer lengths and not observed in the in-domain WSJ test set.
• Despite similar F1 scores on in-domain data, the structured In-Order parser generalizes better to out-of-domain corpora and longer spans, demonstrating that structured prediction enhances robustness.
• The authors achieve state-of-the-art F1 scores on Brown, Genia, and English Web Treebanks, surpassing prior work even without ensembling or self-training.