[Paper Review] End-to-end Semantic Role Labeling with Neural Transition-based Model

Hao Fei, Meishan Zhang+2 more|arXiv (Cornell University)|Jan 2, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper presents the first end-to-end neural transition-based model for semantic role labeling (SRL), jointly performing predicate identification and argument role labeling through incremental transition actions. By employing a close-first parsing order and high-order feature composition to capture long-range dependencies, the model achieves state-of-the-art performance on CoNLL09 and Universal Proposition Bank while maintaining linear-time decoding efficiency.

ABSTRACT

End-to-end semantic role labeling (SRL) has been received increasing interest. It performs the two subtasks of SRL: predicate identification and argument role labeling, jointly. Recent work is mostly focused on graph-based neural models, while the transition-based framework with neural networks which has been widely used in a number of closely-related tasks, has not been studied for the joint task yet. In this paper, we present the first work of transition-based neural models for end-to-end SRL. Our transition model incrementally discovers all sentential predicates as well as their arguments by a set of transition actions. The actions of the two subtasks are executed mutually for full interactions. Besides, we suggest high-order compositions to extract non-local features, which can enhance the proposed transition model further. Experimental results on CoNLL09 and Universal Proposition Bank show that our final model can produce state-of-the-art performance, and meanwhile keeps highly efficient in decoding. We also conduct detailed experimental analysis for a deep understanding of our proposed model.

Motivation & Objective

• To address the limitations of pipeline and graph-based models in end-to-end SRL by introducing a transition-based neural framework.
• To enable mutual interaction between predicate identification and argument role labeling through a unified incremental parsing process.
• To improve long-range dependency modeling by explicitly composing high-order features from previously recognized argument-predicate structures.
• To achieve state-of-the-art performance with high decoding efficiency, outperforming existing graph-based baselines.
• To validate the effectiveness of close-first parsing and high-order feature composition through ablation and analysis.

Proposed method

• The model uses a transition-based parsing framework that incrementally applies actions to identify predicates and assign argument roles in a close-first order.
• Transition decisions are made using a BiLSTM and Stack-LSTM to encode buffer and stack states, with input representations combining word embeddings, character-level features, POS tags, and dependency structures.
• High-order feature composition is introduced to incorporate previously recognized argument-predicate scoring distributions into current action prediction, enhancing long-range dependency modeling.
• The model leverages recursive TreeLSTM to generate dependency-aware representations, improving structural awareness.
• Beam search is used during decoding to refine future decisions, and the system is trained end-to-end with cross-entropy loss.
• Contextualized embeddings (e.g., BERT, ELMo) are integrated to further boost performance.

Experimental results

Research questions

• RQ1 Can a neural transition-based model achieve state-of-the-art performance in end-to-end semantic role labeling?
• RQ2 How does high-order feature composition improve long-distance argument role labeling in a transition-based SRL system?
• RQ3 Does the close-first parsing order enhance the accuracy of argument role labeling, especially for remote arguments?
• RQ4 How does the mutual interaction between predicate identification and argument role labeling affect overall performance?
• RQ5 What is the impact of high-order features and close-first parsing on reducing argument role violations?

Key findings

• The proposed transition-based model achieves state-of-the-art performance on both CoNLL09 and Universal Proposition Bank, outperforming existing graph-based models.
• With high-order feature composition, the model reduces argument role violations by 40% compared to the vanilla model, especially improving continuation and reference role detection.
• The close-first parsing order significantly improves unique core role labeling accuracy, reducing violations by over 50% compared to left-to-right parsing.
• The model maintains linear-time decoding complexity, making it more efficient than graph-based baselines.
• Integration of contextualized embeddings like BERT further improves overall SRL performance, demonstrating the model's compatibility with modern NLP architectures.
• Ablation studies confirm that high-order features are crucial for long-distance argument recognition, with performance drops of up to 15% when removed.