[Paper Review] Transition-Based Dependency Parsing with Stack Long Short-Term Memory

Chris Dyer, Miguel Ballesteros+3 more|Repositori digital de la UPF (Universitat Pompeu Fabra)|May 29, 2015

Natural Language Processing Techniques39 references526 citations

TL;DR

This paper introduces stack LSTMs, a novel recurrent neural network architecture that supports push and pop operations to model parser states in transition-based dependency parsing. By maintaining continuous embeddings of the entire stack, buffer, and action history, the model captures global state information, achieving state-of-the-art performance on English and Chinese dependency parsing benchmarks using end-to-end backpropagation training.

ABSTRACT

We propose a technique for learning representations of parser states in transition-based dependency parsers. Our primary innovation is a new control structure for sequence-to-sequence neural networks---the stack LSTM. Like the conventional stack data structures used in transition-based parsing, elements can be pushed to or popped from the top of the stack in constant time, but, in addition, an LSTM maintains a continuous space embedding of the stack contents. This lets us formulate an efficient parsing model that captures three facets of a parser's state: (i) unbounded look-ahead into the buffer of incoming words, (ii) the complete history of actions taken by the parser, and (iii) the complete contents of the stack of partially built tree fragments, including their internal structures. Standard backpropagation techniques are used for training and yield state-of-the-art parsing performance.

Motivation & Objective

• To address the limitation of prior transition-based parsers that rely on narrow, local views of parser state.
• To model the complete parser state—buffer, stack of partial trees, and action history—using a unified neural representation.
• To develop a differentiable, efficient parsing model that maintains linear time complexity despite global state sensitivity.
• To achieve state-of-the-art dependency parsing accuracy by learning rich, compositional representations of syntactic structures.
• To demonstrate that global context awareness in parsing can be achieved efficiently through a stack-augmented LSTM architecture.

Proposed method

• Proposes stack LSTMs, a variant of LSTMs that support push and pop operations, enabling sequence modeling with a persistent, continuously updated stack memory.
• Uses three separate stack LSTMs to represent the input buffer, the stack of partially built syntactic structures, and the history of parser actions.
• Employs recursive neural networks to compose representations of tree fragments, allowing compositional modeling of syntactic structure.
• Applies standard backpropagation through time for end-to-end training, with loss computed over predicted actions at each parsing step.
• Integrates word embeddings and learns continuous vector representations of the full parser state at each time step.
• Enables efficient, linear-time parsing by incrementally building state representations without reprocessing the entire input.

Experimental results

Research questions

• RQ1 Can a neural network architecture that supports stack-like operations model the full parser state more effectively than local state representations?
• RQ2 Does capturing the complete buffer, stack, and action history lead to improved parsing accuracy in transition-based dependency parsing?
• RQ3 Can a differentiable, recurrent architecture with push and pop operations maintain linear time complexity while modeling global dependencies?
• RQ4 How does the proposed stack LSTM compare to existing neural transition-based parsers in terms of performance and generalization?
• RQ5 To what extent can continuous stack embeddings improve parsing robustness across languages like English and Chinese?

Key findings

• The proposed stack LSTM model achieves state-of-the-art performance on both English and Chinese dependency parsing datasets.
• The model captures long-range dependencies and global state information more effectively than previous methods relying on local context only.
• The use of recursive neural networks for tree fragment composition enables accurate modeling of syntactic structure in partial parse trees.
• End-to-end training via backpropagation yields high-quality representations without manual feature engineering.
• The architecture maintains linear parsing and training time complexity despite modeling the full parser state.
• The model demonstrates that stack-augmented LSTMs can serve as a powerful, differentiable control structure for sequence-to-sequence parsing tasks.