[Paper Review] Joint POS Tagging and Dependency Parsing with Transition-based Neural Networks

Liner Yang, Meishan Zhang+4 more|arXiv (Cornell University)|Apr 25, 2017

Natural Language Processing Techniques16 references3 citations

TL;DR

This paper proposes a joint neural network framework for part-of-speech (POS) tagging and dependency parsing using a transition-based system with three dedicated neural classifiers for shift/reduce decisions, POS tagging, and dependency labeling. The approach significantly outperforms prior methods across eight languages on three treebanks by leveraging end-to-end learning to overcome feature sparsity and incompleteness in traditional models.

ABSTRACT

While part-of-speech (POS) tagging and dependency parsing are observed to be closely related, existing work on joint modeling with manually crafted feature templates suffers from the feature sparsity and incompleteness problems. In this paper, we propose an approach to joint POS tagging and dependency parsing using transition-based neural networks. Three neural network based classifiers are designed to resolve shift/reduce, tagging, and labeling conflicts. Experiments show that our approach significantly outperforms previous methods for joint POS tagging and dependency parsing across a variety of natural languages.

Motivation & Objective

• To address the limitations of manually crafted feature templates in joint POS tagging and dependency parsing, which suffer from sparsity and incompleteness.
• To improve joint performance by enabling lexical and syntactic information to interact dynamically during parsing, rather than through static feature engineering.
• To design a unified transition-based framework where POS tagging and dependency parsing are jointly optimized via neural classifiers.
• To reduce error propagation and improve tagging accuracy by allowing parsing to inform tagging decisions through shared representations.

Proposed method

• Extends a transition-based parsing system with five actions: Shift, Left, Right, Tag t, and Label l, to jointly model POS tagging and dependency parsing.
• Employs three separate neural classifiers: one for shift/reduce decisions, one for POS tagging, and one for dependency labeling, each using dense vector representations instead of sparse features.
• Uses bidirectional LSTMs to encode word representations and concatenates contextual features (e.g., stack, buffer, and tag history) for each classifier input.
• Applies multi-layer perceptrons with ReLU activation to map input representations to action scores, enabling end-to-end training.
• Integrates word embeddings and hidden states across tagging and parsing components to allow mutual supervision and error reduction.
• Employs a joint decoding strategy that selects actions based on the highest-scoring combination of parsing, tagging, and labeling decisions at each step.

Experimental results

Research questions

• RQ1 Can a transition-based neural network jointly model POS tagging and dependency parsing more effectively than traditional feature-based models?
• RQ2 Does end-to-end learning with neural classifiers reduce the impact of feature sparsity and incompleteness in joint parsing?
• RQ3 How does the interaction between tagging and parsing improve performance compared to cascaded or independent modeling?
• RQ4 What is the relative contribution of syntactic information to tagging and lexical information to parsing in the joint framework?

Key findings

• The proposed method achieves state-of-the-art performance on joint POS tagging and dependency parsing across three treebanks and eight natural languages.
• The model significantly outperforms previous joint models that rely on manually designed feature templates, particularly in handling ambiguous tagging cases.
• Disabling interactions between tagging and parsing leads to a substantial drop in both tagging and parsing accuracy, confirming the benefit of joint learning.
• Providing tagging information to parsing (tag → parse) yields greater gains than providing parsing information to tagging (tag ← parse), due to the local nature of tagging ambiguity versus the long-range dependency needs of parsing.
• The model maintains strong performance even when syntactic features are removed from the tagging classifier, indicating robustness through shared representations.
• Ablation studies confirm that the joint architecture reduces error propagation and improves generalization, especially in low-resource settings.