[Paper Review] Syntax-aware Neural Semantic Role Labeling with Supertags

Jungo Kasai, Dan Friedman+3 more|arXiv (Cornell University)|Mar 12, 2019

Natural Language Processing Techniques28 references4 citations

TL;DR

This paper proposes a syntax-aware neural semantic role labeling (SRL) model that uses supertags—rich syntactic tags derived from dependency parses—instead of full syntactic trees. By training a BiLSTM to predict supertags and feeding them alongside word and POS embeddings into a deep highway BiLSTM for SRL, the model achieves state-of-the-art performance on CoNLL 2009 English and Spanish datasets, outperforming both syntax-agnostic and full-parse-based models.

ABSTRACT

We introduce a new syntax-aware model for dependency-based semantic role labeling that outperforms syntax-agnostic models for English and Spanish. We use a BiLSTM to tag the text with supertags extracted from dependency parses, and we feed these supertags, along with words and parts of speech, into a deep highway BiLSTM for semantic role labeling. Our model combines the strengths of earlier models that performed SRL on the basis of a full dependency parse with more recent models that use no syntactic information at all. Our local and non-ensemble model achieves state-of-the-art performance on the CoNLL 09 English and Spanish datasets. SRL models benefit from syntactic information, and we show that supertagging is a simple, powerful, and robust way to incorporate syntax into a neural SRL system.

Motivation & Objective

• To improve semantic role labeling (SRL) performance by incorporating syntactic information without relying on full dependency parsing.
• To investigate whether supertags—linguistically rich, sequence-level tags—can serve as an effective and robust alternative to full syntactic parses in neural SRL.
• To evaluate different supertag designs for their impact on SRL performance across varying sentence lengths and predicate types.
• To demonstrate that supertagging offers a middle ground between syntax-agnostic models and complex parse-based models, improving generalization and robustness.
• To explore the utility of supertags in low-resource or out-of-domain settings where full parsing may fail.

Proposed method

• Extract supertags from dependency parses of the CoNLL 2009 corpus, using four distinct supertag models: Model 0 (relation + direction), Model 1 (adds info on obligatory dependents), Model 2 (adds relations to obligatory dependents), and a new Model TAG inspired by Tree-Adjoining Grammar.
• Train a bidirectional LSTM (BiLSTM) supertagger to predict supertags from words and parts of speech, using real-valued embeddings for the predicted supertags.
• Feed the supertag embeddings, along with word and POS embeddings, into a deep highway BiLSTM for semantic role labeling.
• Use a non-ensemble, local model architecture to ensure efficiency and avoid error propagation from parsing.
• Evaluate performance on CoNLL 2009 English and Spanish in-domain and out-of-domain test sets, comparing across supertag models and baselines.
• Analyze performance by sentence length and predicate category to assess robustness and linguistic sensitivity.

Experimental results

Research questions

• RQ1 Can supertags derived from dependency parses improve SRL performance compared to syntax-agnostic neural models?
• RQ2 How do different supertag designs—varying in syntactic expressiveness—affect SRL performance on English and Spanish?
• RQ3 Does using supertags provide a more robust and generalizable alternative to full dependency parsing in SRL?
• RQ4 In what ways do supertags enhance SRL performance on longer or out-of-domain sentences?
• RQ5 Can supertags outperform more complex syntactic encoding methods, such as dependency path modeling?

Key findings

• The proposed model with Model 1 supertags achieves state-of-the-art F1 scores of 92.0 on the CoNLL 2009 English in-domain test set, outperforming the BiLSTM baseline and Path-LSTM model.
• Model 1 supertags improve performance across all predicate categories (V/A0, V/A1, V/A2, V/AM, N/A2, N/AM), with consistent gains over the BiLSTM and BDH baselines.
• The model achieves an F1 score of 92.9 on the Spanish in-domain test set, demonstrating strong cross-lingual generalization.
• Supertag-based models show greater gains on longer sentences, especially in out-of-domain settings, indicating robustness to structural complexity.
• Model 0 (simplest supertag) outperforms the Path-LSTM model in key roles (V/A0, V/A1, V/AM, N/AM), showing that even basic supertags can surpass more complex syntactic path modeling.
• The model achieves SOTA results without ensembling and without relying on full syntactic parsing, demonstrating the effectiveness of supertags as a lightweight, syntax-rich signal.