[Paper Review] Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation

Goran Glavašš, Ivan Vulić|arXiv (Cornell University)|Aug 15, 2020

Natural Language Processing Techniques10 citations

TL;DR

This paper investigates whether supervised syntactic parsing, injected via intermediate parsing training (IPT) into RoBERTa and XLM-R transformers, improves performance on downstream language understanding (LU) tasks. Despite achieving state-of-the-art parsing results, IPT yields negligible and inconsistent gains on NLI, paraphrase identification, and causal commonsense reasoning, suggesting that explicit Universal Dependencies syntax provides little benefit for LU in the era of large-scale language model pretraining.

ABSTRACT

Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). The recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, however, questions this belief. In this work, we empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks. Relying on the established fine-tuning paradigm, we first couple a pretrained transformer with a biaffine parsing head, aiming to infuse explicit syntactic knowledge from Universal Dependencies treebanks into the transformer. We then fine-tune the model for LU tasks and measure the effect of the intermediate parsing training (IPT) on downstream LU task performance. Results from both monolingual English and zero-shot language transfer experiments (with intermediate target-language parsing) show that explicit formalized syntax, injected into transformers through IPT, has very limited and inconsistent effect on downstream LU performance. Our results, coupled with our analysis of transformers' representation spaces before and after intermediate parsing, make a significant step towards providing answers to an essential question: how (un)availing is supervised parsing for high-level semantic natural language understanding in the era of large neural models?

Motivation & Objective

• To assess whether supervised syntactic parsing, injected through intermediate parsing training (IPT), enhances downstream language understanding (LU) performance in state-of-the-art transformer models.
• To investigate whether explicit syntactic knowledge from Universal Dependencies (UD) treebanks improves LU tasks such as natural language inference, paraphrase identification, and causal commonsense reasoning.
• To compare the impact of IPT with direct fine-tuning and other methods like in-context learning or adapter-based tuning, focusing on representation changes in transformer layers.
• To analyze whether the structural knowledge from supervised parsing aligns with the type of syntactic generalizations needed for high-level semantic understanding in neural models.
• To explore the role of syntactic inductive bias in low-resource settings versus high-resource, pretraining-heavy setups, especially in zero-shot multilingual transfer.

Proposed method

• Fine-tune RoBERTa and XLM-R with a biaffine dependency parsing head to perform intermediate parsing training (IPT) on Universal Dependencies (UD) treebanks.
• Train the model end-to-end for dependency parsing using standard cross-entropy loss on gold-standard UD annotations, achieving SOTA parsing performance.
• Fine-tune the same syntactically-informed transformer models on three downstream LU tasks: natural language inference (MNLI), paraphrase identification (SNLI), and causal commonsense reasoning (HellaSwag).
• Compare downstream performance of IPT-fine-tuned models against directly fine-tuned baselines (without IPT) to isolate the effect of injected syntactic knowledge.
• Use linear-CKA (centered kernel alignment) to analyze representation changes in transformer layers before and after IPT, measuring the degree of new syntactic information injected.
• Conduct both monolingual (English) and zero-shot multilingual transfer experiments, where the model is IPT-fine-tuned on target language UD treebanks before being evaluated on English LU tasks.

Experimental results

Research questions

• RQ1 Does intermediate parsing training (IPT) with Universal Dependencies (UD) syntax improve downstream language understanding (LU) performance in RoBERTa and XLM-R?
• RQ2 How does IPT affect the representation space of transformer layers, and does it inject meaningful syntactic knowledge not already present via language modeling pretraining?
• RQ3 Is the syntactic information from supervised parsing aligned with the structural generalizations required for high-level semantic understanding in LU tasks?
• RQ4 How does the effectiveness of IPT vary across different models (e.g., BERT vs. RoBERTa) and in zero-shot multilingual transfer settings?
• RQ5 To what extent does IPT provide new inductive bias compared to direct fine-tuning or other methods like adapters or in-context learning?

Key findings

• Intermediate parsing training (IPT) with Universal Dependencies (UD) syntax yields negligible and inconsistent improvements on downstream language understanding tasks, including natural language inference (MNLI), paraphrase identification (SNLI), and causal commonsense reasoning (HellaSwag).
• Despite achieving state-of-the-art parsing performance, IPT provides minimal benefit to downstream LU tasks, suggesting that explicit syntactic supervision is largely unavailing for high-level understanding in large-scale pretraining paradigms.
• Linear-CKA analysis shows that IPT injects more new syntactic information than direct fine-tuning (ILMT), especially for BERT, but this does not translate into performance gains, indicating a mismatch between injected syntax and task-relevant structure.
• The l-CKA scores in lower layers are very high between IPT and original BERT, suggesting that the syntactic knowledge from UD treebanks is redundant with respect to the structural knowledge already learned via language modeling pretraining.
• RoBERTa shows significantly less parameter change after IPT than BERT, indicating that RoBERTa’s representations are already richer and more robust, possibly due to larger-scale and longer pretraining.
• In zero-shot multilingual transfer, IPT injects more syntactic knowledge into mBERT and XLM-R for low-resource languages (e.g., German, Turkish), but this again leads to mostly insignificant performance gains, reinforcing the conclusion that supervised parsing is not beneficial for LU in this setup.