[Paper Review] Discriminative Neural Sentence Modeling by Tree-Based Convolution

Lili Mou, Hao Peng+4 more|arXiv (Cornell University)|Apr 5, 2015

Topic Modeling36 references65 citations

TL;DR

This paper proposes Tree-Based Convolutional Neural Networks (TBCNNs) that leverage constituency or dependency parsing trees to extract structural features via tree-structured convolutions, enabling short propagation paths for effective feature learning. TBCNNs achieve state-of-the-art performance on sentiment analysis and question classification, outperforming prior neural networks and handcrafted feature methods.

ABSTRACT

This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling. Our models leverage either constituency trees or dependency trees of sentences. The tree-based convolution process extracts sentences' structural features, and these features are aggregated by max pooling. Such architecture allows short propagation paths between the output layer and underlying feature detectors, which enables effective structural feature learning and extraction. We evaluate our models on two tasks: sentiment analysis and question classification. In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering. We also make efforts to visualize the tree-based convolution process, shedding light on how our models work.

Motivation & Objective

• To develop a neural network architecture that effectively captures syntactic structure in sentences for discriminative sentence modeling.
• To address the limitations of standard CNNs (lack of structural awareness) and RNNs (long propagation paths in deep trees) in sentence representation learning.
• To design a model that combines the benefits of short propagation paths (like CNNs) and structural feature learning (like RNNs) using parsing trees.
• To evaluate the model on real-world NLP tasks and demonstrate superior performance over existing methods.
• To provide interpretability through visualization of feature activation and pooling processes.

Proposed method

• The model uses either constituency trees (c-TBCNN) or dependency trees (d-TBCNN) as the structural backbone for sentence representation.
• Tree-based convolution applies fixed-size window feature detectors over subtrees, with each window sliding over nodes in the parsing tree.
• Feature vectors from each convolution window are computed using a weight matrix and activation function: $\bm{y} = f(W \cdot [\bm{x}_1; \cdots; \bm{x}_t] + \bm{b})$, where $\bm{x}_i$ are word embeddings.
• Max pooling aggregates the highest activation across all nodes for each feature dimension, producing a fixed-size representation.
• The final sentence representation is fed into a classifier for downstream tasks like sentiment analysis and question classification.
• The architecture ensures all features have short paths to the output layer, enabling efficient learning of structural dependencies.

Experimental results

Research questions

• RQ1 Can tree-structured convolutions effectively extract syntactic and semantic features from sentences while maintaining short information propagation paths?
• RQ2 Does leveraging parsing trees (constituency or dependency) improve sentence modeling performance compared to flat CNNs or recursive RNNs?
• RQ3 How do TBCNNs compare to state-of-the-art models in sentiment analysis and question classification tasks?
• RQ4 To what extent can the internal feature learning process of TBCNNs be interpreted and visualized?
• RQ5 Does the model generalize well across varying sentence lengths, especially long sentences?

Key findings

• d-TBCNN outperformed previous state-of-the-art results on both sentiment analysis and question classification tasks.
• c-TBCNN and d-TBCNN consistently outperformed a re-implemented RNN across all sentence length groups, with the performance gap increasing for sentences longer than 20 words.
• The model achieved higher accuracy than the RNN baseline, which was re-implemented to ensure fair comparison (43.2% vs. 42.7%).
• Visualization showed that features from semantically meaningful phrases—such as "the stunning dreamlike visual"—had higher pooling fractions (0.19), indicating their relevance to the task.
• Stop words like "the" and "will" contributed minimal features to the pooling layer, consistent with their low semantic informativeness.
• The model demonstrated that window-level features integrating multiple words (e.g., "the stunning dreamlike visual") were more discriminative than isolated word windows, even when the root node was sentiment-neutral.