[Paper Review] Cutting Recursive Autoencoder Trees

Christian Scheible, Hinrich Schuetze|arXiv (Cornell University)|Jan 13, 2013

Topic Modeling9 references3 citations

TL;DR

This paper investigates the interpretability and structural efficiency of Semi-Supervised Recursive Autoencoders (RAEs) in sentiment classification. It proposes automatic pruning of RAE tree structures and evaluates them via classification accuracy and human judgment. The key finding is that word embeddings alone achieve state-of-the-art performance, while the induced tree structures contribute little beyond what embeddings already capture, suggesting limited syntactic or semantic interpretability.

ABSTRACT

Deep Learning models enjoy considerable success in Natural Language Processing. While deep architectures produce useful representations that lead to improvements in various tasks, they are often difficult to interpret. This makes the analysis of learned structures particularly difficult. In this paper, we rely on empirical tests to see whether a particular structure makes sense. We present an analysis of the Semi-Supervised Recursive Autoencoder, a well-known model that produces structural representations of text. We show that for certain tasks, the structure of the autoencoder can be significantly reduced without loss of classification accuracy and we evaluate the produced structures using human judgment.

Motivation & Objective

• To assess whether the syntactic and semantic structures produced by Recursive Autoencoders (RAEs) are interpretable in natural language tasks.
• To investigate whether RAE tree structures can be simplified without sacrificing classification accuracy.
• To determine the relative contribution of RAE-learned tree structures versus word embeddings in sentiment classification.
• To evaluate the interpretability of RAE structures through human annotation on syntactic and semantic criteria.

Proposed method

• The authors apply multiple structural simplification techniques: level cuts (limiting tree depth), subtree selection (choosing specific subtrees), and window selection (varying context window size around target words).
• They train a Semi-Supervised Recursive Autoencoder (RAE) on a sentiment classification dataset, using both reconstruction and classification objectives with a combined loss function.
• The model uses word embeddings that are jointly optimized with the autoencoder parameters, and final representations are derived from both the embeddings and the RAE’s hierarchical node representations.
• For evaluation, they compare classification accuracy across different pruning strategies and assess human judgment on the linguistic plausibility of the resulting structures.
• They conduct ablation studies by removing embeddings or tree structures entirely to isolate their contributions.
• They use a window-based feature extraction method where the context window size is varied to test the impact of local context on performance.

Experimental results

Research questions

• RQ1 Can the structural representations learned by RAEs be meaningfully interpreted as syntactic or semantic trees?
• RQ2 To what extent can RAE tree structures be simplified (e.g., via pruning or windowing) without loss of classification accuracy?
• RQ3 What is the relative contribution of word embeddings versus RAE-induced tree structures to classification performance?
• RQ4 Do human annotators perceive the RAE-generated structures as linguistically coherent in terms of syntax and semantics?

Key findings

• The best-performing model uses only word embeddings and achieves state-of-the-art accuracy, outperforming models that rely on full RAE tree structures.
• Structural pruning via level cuts or subtree selection leads to significant accuracy drops unless the full tree is used, indicating that the tree structure itself does not improve performance.
• Window selection experiments show that a context window of 15 words (in each direction) is sufficient for high performance, suggesting that local context is more important than hierarchical structure.
• Human evaluation reveals that RAE-generated structures are not consistently interpretable as syntactic or semantic trees, indicating weak alignment with linguistic categories.
• Combining word embeddings with RAE-learned representations provides no performance gain, implying high redundancy between the two representations.
• Models trained with strict depth limits (e.g., max level 1) perform poorly, but performance recovers only when full trees are used, suggesting error propagation and dependency in full structures are critical for optimization.