[Paper Review] Long Short-Term Memory Over Tree Structures

Xiaodan Zhu, Parinaz Sobhani+1 more|arXiv (Cornell University)|Mar 16, 2015

Topic Modeling26 references68 citations

TL;DR

This paper proposes S-LSTM, a novel extension of Long Short-Term Memory (LSTM) that generalizes the recurrent memory mechanism to tree-structured data, enabling memory cells to aggregate information from multiple child and descendant nodes via gated memory copying. The model significantly outperforms state-of-the-art recursive neural networks on sentiment analysis tasks by better modeling long-distance dependencies in hierarchical structures, achieving 44.1% accuracy on the Stanford Sentiment Tree Bank with full node annotations.

ABSTRACT

The chain-structured long short-term memory (LSTM) has showed to be effective in a wide range of problems such as speech recognition and machine translation. In this paper, we propose to extend it to tree structures, in which a memory cell can reflect the history memories of multiple child cells or multiple descendant cells in a recursive process. We call the model S-LSTM, which provides a principled way of considering long-distance interaction over hierarchies, e.g., language or image parse structures. We leverage the models for semantic composition to understand the meaning of text, a fundamental problem in natural language understanding, and show that it outperforms a state-of-the-art recursive model by replacing its composition layers with the S-LSTM memory blocks. We also show that utilizing the given structures is helpful in achieving a performance better than that without considering the structures.

Motivation & Objective

• To address the limitation of chain-structured LSTMs in modeling hierarchical, non-linear structures common in natural language and vision.
• To extend the LSTM architecture to tree-structured recursive networks to better capture long-distance dependencies in syntactic and semantic hierarchies.
• To empirically evaluate whether explicit structural information improves performance in semantic composition tasks compared to implicit structure learning via linear chains.
• To demonstrate that S-LSTM can outperform state-of-the-art recursive neural networks by replacing their composition layers with learnable, gated memory blocks.

Proposed method

• Proposes S-LSTM, a tree-structured variant of LSTM where each node’s memory cell aggregates memory vectors from multiple children using learnable gates.
• Adapts the standard LSTM cell equations to support multiple input memory vectors from child nodes, with separate input, forget, and output gates per child.
• Uses a recursive bottom-up computation: for each internal node, the hidden and cell states are computed based on the weighted combination of child states, enabling hierarchical memory propagation.
• Employs backpropagation through structures to train the model end-to-end, with gradients computed over the tree topology.
• Replaces the tensor-enhanced composition layer in a state-of-the-art recursive neural network with the S-LSTM memory block to enable direct comparison.
• Evaluates the model on the Stanford Sentiment Tree Bank, using both root-level and full-tree node annotations for training and evaluation.

Experimental results

Research questions

• RQ1 Can an LSTM-based architecture be effectively extended to tree-structured data to model hierarchical dependencies more effectively than standard recursive neural networks?
• RQ2 Does explicitly modeling tree structures lead to improved performance in semantic composition tasks compared to learning structures implicitly via linear chains?
• RQ3 To what extent does S-LSTM mitigate the vanishing gradient problem in deep hierarchical structures compared to traditional recursive networks?
• RQ4 How does the performance of S-LSTM vary across different depths and phrase lengths in syntactic trees?

Key findings

• S-LSTM achieves 44.1% accuracy on the Stanford Sentiment Tree Bank when using both root and leaf node annotations, outperforming the previous state-of-the-art recursive model.
• The model significantly improves performance at deeper levels of the parse tree, demonstrating superior modeling of complex syntax and semantics in longer phrases.
• When no structural information is provided, even left- or right-recursive S-LSTM variants achieve only 40.2% and 40.3% accuracy (using only root labels), showing that explicit structure is crucial for optimal performance.
• The performance gap between S-LSTM and non-structured variants is most pronounced when using only root-level labels, indicating that structural priors are especially valuable when supervision is limited.
• S-LSTM consistently outperforms all baselines across all depth and length levels in the tree, confirming its ability to model long-range dependencies in hierarchical representations.
• The results provide empirical evidence that explicitly utilizing parse structures leads to better semantic representations than relying on linear-chain RNNs to implicitly capture structure.