[Paper Review] When Are Tree Structures Necessary for Deep Learning of Representations?

Jiwei Li, Minh-Thang Luong+2 more|arXiv (Cornell University)|Feb 28, 2015

Topic Modeling46 references135 citations

TL;DR

This paper investigates when recursive neural networks (tree-structured models) offer advantages over simpler recurrent neural networks (RNNs) for learning text representations. By comparing tree-based and sequence-based models across four NLP tasks, the authors find that recursive models significantly outperform RNNs only on tasks requiring long-distance semantic dependencies—particularly semantic relation extraction. They further show that a simple heuristic—splitting long sentences at punctuation into clause-like units before processing—can make standard RNNs achieve performance comparable to recursive models, suggesting that structural decomposition, not syntactic parsing per se, is the key factor in their success.

ABSTRACT

Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. But there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper we benchmark {\bf recursive} neural models against sequential {\bf recurrent} neural models (simple recurrent and LSTM models), enforcing apples-to-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase level; (2) matching questions to answer-phrases; (3) discourse parsing; (4) semantic relation extraction (e.g., {\em component-whole} between nouns). Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require associating headwords across a long distance, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.

Motivation & Objective

• To determine under what conditions recursive neural models outperform simpler recurrent models in learning text representations.
• To evaluate whether syntactic parse trees are truly necessary for improved performance on NLP tasks.
• To investigate whether recurrent models can be enhanced to match recursive models' performance through architectural modifications.
• To understand the role of structural decomposition in handling long-distance dependencies in text.

Proposed method

• Benchmarking recursive neural models (using syntactic parse trees) against sequential RNNs (including LSTM and bidirectional variants) across four NLP tasks with controlled, apples-to-apples comparisons.
• Applying the same training setup (AdaGrad, mini-batches, initialization) to ensure fairness in model comparison.
• Introducing a heuristic method to improve RNNs: splitting long sentences at punctuation into clause-like units, processing them separately, then combining representations.
• Using fixed-size vector representations from both models as input to downstream classifiers for sentiment, matching, discourse, and relation classification tasks.
• Evaluating model performance on standardized datasets: Stanford Sentiment Treebank, UMD-QA, SemEval-2010, and RST discourse parsing.
• Analyzing intermediate representations (node outputs in trees, time-step outputs in RNNs) to assess linguistic meaningfulness and information content.

Experimental results

Research questions

• RQ1 In which NLP tasks do recursive neural models significantly outperform recurrent neural networks?
• RQ2 Is the performance gain of recursive models due to syntactic structure, or due to structural decomposition of long sequences?
• RQ3 Can recurrent models be made to match recursive model performance through architectural modifications?
• RQ4 Do intermediate representations in RNNs capture linguistically meaningful information comparable to parse tree nodes?
• RQ5 How does supervision quality and sequence length affect the relative performance of tree-based versus sequence-based models?

Key findings

• Recursive neural models significantly outperform RNNs only on tasks requiring long-distance semantic dependencies, such as semantic relation extraction between headwords separated by long spans.
• On the SemEval-2010 dataset, recursive models achieved a 3.5% absolute improvement in F1 over the best RNN baseline, highlighting their advantage in capturing long-range relations.
• For long sequences with insufficient supervision (e.g., only top-level labels), no significant performance difference was observed between recursive and recurrent models.
• A simple clause-splitting heuristic—breaking long sentences at punctuation and processing units separately—enabled RNNs to match recursive model performance on sentiment classification, suggesting structural decomposition is key.
• Bidirectional RNNs largely closed the performance gap on shorter sequences, indicating that sequential modeling with bidirectional context can compensate for lack of syntactic structure.
• Despite lacking linguistic interpretability, intermediate RNN outputs (at different time steps) performed as well as linguistically meaningful parse tree nodes in the UMD-QA phrase matching task.