[Paper Review] Parsing as Pretraining

David Vilares, Michalina Strzyz+2 more|arXiv (Cornell University)|Feb 5, 2020

Topic Modeling34 references6 citations

TL;DR

This paper proposes a novel method to perform full constituent and dependency parsing using only pretrained language model encoders, without any task-specific decoders or parsing algorithms. By reformulating parsing as sequence labeling and using a single feed-forward layer to map word vectors to linearized tree structures, the approach achieves state-of-the-art results—93.5% bracketing F1 on PTB and 78.8% LAS on en-ewt UD—demonstrating that pretrained models inherently encode syntactic structure even without fine-tuning.

ABSTRACT

Recent analyses suggest that encoders pretrained for language modeling capture certain morpho-syntactic structure. However, probing frameworks for word vectors still do not report results on standard setups such as constituent and dependency parsing. This paper addresses this problem and does full parsing (on English) relying only on pretraining architectures -- and no decoding. We first cast constituent and dependency parsing as sequence tagging. We then use a single feed-forward layer to directly map word vectors to labels that encode a linearized tree. This is used to: (i) see how far we can reach on syntax modelling with just pretrained encoders, and (ii) shed some light about the syntax-sensitivity of different word vectors (by freezing the weights of the pretraining network during training). For evaluation, we use bracketing F1-score and LAS, and analyze in-depth differences across representations for span lengths and dependency displacements. The overall results surpass existing sequence tagging parsers on the PTB (93.5%) and end-to-end EN-EWT UD (78.8%).

Motivation & Objective

• To investigate whether pretrained language models alone can perform full parsing without any task-specific decoding or structural features.
• To assess the syntax-sensitivity of different word vector representations by freezing pretraining network weights during training.
• To evaluate how well various pretraining architectures capture morpho-syntactic structures through direct parsing performance.
• To provide a unified framework for analyzing syntactic abilities of word vectors using full parsing setups.

Proposed method

• Casting both constituent and dependency parsing as sequence labeling tasks by linearizing tree structures into label sequences.
• Using a single feed-forward layer to map input word embeddings to output label sequences representing parsed trees.
• Freezing the pretraining encoder weights to analyze the syntax-sensitivity of static word vectors, and fine-tuning them to evaluate end-to-end performance.
• Evaluating on standard benchmarks: PTB for constituency parsing and en-ewt UD for dependency parsing using bracketing F1 and LAS scores.
• Analyzing performance across span lengths, span labels, dependency displacements, and relation types to probe linguistic capabilities.
• Comparing precomputed embeddings (e.g., FastText) with contextualized representations (e.g., ELMo, BERT) under identical setup.

Experimental results

Research questions

• RQ1 To what extent can pretrained language models perform full parsing without any task-specific decoding or structural features?
• RQ2 Which pretraining architectures encode the most syntactic structure, as measured by parsing performance on frozen weights?
• RQ3 How do different word vector types (static vs. contextualized) perform across syntactic phenomena such as span length, label frequency, and dependency displacement?
• RQ4 Do language modeling objectives inherently encode phrase-structure more than dependency structure?
• RQ5 Can sequence labeling with a single linear head achieve competitive parsing performance using only pretrained encoders?

Key findings

• The model achieves a bracketing F1 score of 93.5% on the PTB test set and 78.8% LAS on the en-ewt UD test set, surpassing existing sequence tagging parsers.
• Frozen ELMo and BERT representations outperform static FastText embeddings on span identification, especially for longer and less frequent spans.
• Contextualized representations (ELMo, BERT) show significantly better performance on complex syntactic structures like long or rare spans and difficult dependency relations such as root, nsubj, and obj.
• Performance on dependency parsing is lower than on constituency parsing, with ELMo and BERT showing smaller improvements over static vectors, suggesting weaker encoding of dependency structures.
• The results indicate that language modeling objectives may naturally encode phrase-structure concepts more effectively than dependency relations, as evidenced by better performance on constituent parsing and span-level tasks.
• The analysis reveals that BERT, when fine-tuned, is capable of achieving strong parsing performance solely through its pretrained representations, without any additional architectural design for parsing.