[Paper Review] A C-LSTM Neural Network for Text Classification

Chunting Zhou, Chonglin Sun+2 more|arXiv (Cornell University)|Nov 27, 2015

Topic Modeling19 references657 citations

TL;DR

This paper proposes C-LSTM, a unified neural network architecture that combines convolutional neural networks (CNNs) for local phrase feature extraction with long short-term memory (LSTM) networks to model sequential dependencies in sentence representations. By feeding CNN-generated higher-level n-gram features into an LSTM, the model captures both local phrase semantics and global sentence structure, achieving state-of-the-art or competitive performance on sentiment classification and question classification tasks without requiring external linguistic annotations.

ABSTRACT

Neural network models have been demonstrated to be capable of achieving remarkable performance in sentence and document modeling. Convolutional neural network (CNN) and recurrent neural network (RNN) are two mainstream architectures for such modeling tasks, which adopt totally different ways of understanding natural languages. In this work, we combine the strengths of both architectures and propose a novel and unified model called C-LSTM for sentence representation and text classification. C-LSTM utilizes CNN to extract a sequence of higher-level phrase representations, and are fed into a long short-term memory recurrent neural network (LSTM) to obtain the sentence representation. C-LSTM is able to capture both local features of phrases as well as global and temporal sentence semantics. We evaluate the proposed architecture on sentiment classification and question classification tasks. The experimental results show that the C-LSTM outperforms both CNN and LSTM and can achieve excellent performance on these tasks.

Motivation & Objective

• To address the limitations of standalone CNNs and RNNs in capturing both local phrase features and long-term sequential dependencies in text.
• To develop an end-to-end, unified architecture that leverages the strengths of both CNN and LSTM without relying on syntactic parse trees or external linguistic knowledge.
• To improve sentence representation for downstream tasks like sentiment classification and question type classification through hierarchical feature learning.
• To evaluate whether learning from higher-level representations (e.g., n-grams) enhances LSTM’s ability to model long-range dependencies.

Proposed method

• A single-layer CNN is applied to pre-trained word vectors to extract local n-gram features, producing higher-level phrase representations.
• The output feature maps from the CNN are organized as a sequence of higher-level representations, preserving their order for sequential modeling.
• These sequential higher-level features are fed into an LSTM to capture long-term dependencies and global sentence semantics.
• The entire model is trained end-to-end in an end-to-end fashion, with no need for external linguistic preprocessing or syntactic parsing.
• Multiple filter configurations are evaluated, including single-layer with fixed filter size and multi-layer with varying filter lengths.
• Hyperparameters are tuned via grid search, and performance is evaluated on benchmark datasets for sentiment and question classification.

Experimental results

Research questions

• RQ1 Can combining CNN and LSTM in a unified architecture improve sentence representation for text classification tasks?
• RQ2 Does learning from higher-level representations (n-grams) enhance the LSTM’s ability to model long-term dependencies compared to direct input from word sequences?
• RQ3 How does the choice of filter size in the CNN layer affect model performance on local feature extraction?
• RQ4 Can the C-LSTM model achieve competitive results without relying on hand-engineered features or syntactic parse trees?

Key findings

• C-LSTM outperforms both standalone CNN and LSTM models on both sentiment classification and 6-way question classification tasks.
• The model achieves the fourth-best published result on the 5-class sentiment classification task, demonstrating strong performance despite being end-to-end and free of linguistic annotations.
• For binary sentiment classification, C-LSTM achieves results comparable to state-of-the-art models, including those relying on extensive feature engineering.
• The single-layer CNN with filter length 3 consistently outperforms other configurations, indicating that trigram features are particularly effective for the tasks studied.
• The model’s performance is close to that of an SVM with highly engineered features (e.g., POS tags, WordNet, parser outputs), but without requiring human-designed features or error-prone NLP tools.
• The results confirm that LSTM benefits significantly from learning sequential patterns in higher-level representations, suggesting that hierarchical feature abstraction improves sequence modeling.