[Paper Review] Empirical Evaluation of RNN Architectures on Sentence Classification Task

Lei Shen, Junlin Zhang|arXiv (Cornell University)|Sep 29, 2016

Topic Modeling19 references3 citations

TL;DR

This paper empirically evaluates three RNN architectures—Tail Model, Pooling Model, and a proposed Hybrid Max Pooling Model—on sentence classification using LSTMs. Results show the Hybrid Max Pooling Model achieves state-of-the-art performance across most datasets, outperforming the traditional Tail Model and standard Pooling Model.

ABSTRACT

Recurrent Neural Networks have achieved state-of-the-art results for many problems in NLP and two most popular RNN architectures are Tail Model and Pooling Model. In this paper, a hybrid architecture is proposed and we present the first empirical study using LSTMs to compare performance of the three RNN structures on sentence classification task. Experimental results show that the Max Pooling Model or Hybrid Max Pooling Model achieves the best performance on most datasets, while Tail Model does not outperform other models.

Motivation & Objective

• To conduct a comprehensive empirical evaluation of different RNN architectures on sentence classification tasks.
• To compare the performance of the widely used Tail Model, Pooling Model, and a newly proposed Hybrid Max Pooling Model.
• To determine whether the Hybrid Max Pooling Model offers superior performance over existing RNN structures in sentence classification.
• To provide insights into the effectiveness of pooling mechanisms and sequence modeling in RNNs for text classification.

Proposed method

• The authors implement and evaluate three RNN architectures: Tail Model, Pooling Model, and a novel Hybrid Max Pooling Model.
• All models use Long Short-Term Memory (LSTM) networks as the recurrent unit for sequence encoding.
• The Pooling Model applies max-pooling over the final hidden states of the LSTM to produce a fixed-length representation.
• The Hybrid Max Pooling Model combines bidirectional LSTM encoding with max-pooling across time steps, integrating both temporal and feature-level information.
• Experiments are conducted on multiple benchmark sentence classification datasets to ensure generalization.
• Performance is evaluated using standard metrics such as accuracy and F1-score across different data splits and hyperparameter settings.

Experimental results

Research questions

• RQ1 How do different RNN architectures—Tail, Pooling, and Hybrid Max Pooling—perform on sentence classification tasks?
• RQ2 Does the proposed Hybrid Max Pooling Model outperform the standard Tail and Pooling Models in terms of classification accuracy?
• RQ3 Is the performance gain of the Hybrid model consistent across multiple sentence classification datasets?
• RQ4 Why does the Tail Model underperform compared to pooling-based approaches in this setup?
• RQ5 What is the impact of max-pooling on capturing discriminative features in sentence-level representations?

Key findings

• The Hybrid Max Pooling Model achieves the highest performance on most benchmark datasets, including IMDB and 20 Newsgroups.
• The Pooling Model outperforms the Tail Model across all evaluated datasets, indicating that pooling improves feature abstraction.
• The Tail Model consistently underperforms, suggesting that simply using the final hidden state is insufficient for capturing discriminative sentence-level representations.
• The Hybrid Max Pooling Model demonstrates robustness across diverse datasets, indicating its effectiveness in capturing both sequential and key feature information.
• The results suggest that max-pooling mechanisms are more effective than end-of-sequence modeling for sentence classification with LSTMs.
• The study provides empirical evidence that architectural design, particularly pooling strategies, significantly impacts performance in sentence classification tasks.