[Paper Review] Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Peng Zhou, Zhenyu Qi+4 more|arXiv (Cornell University)|Nov 21, 2016

Sentiment Analysis and Opinion Mining39 references305 citations

TL;DR

This paper introduces BLSTM-2DPooling and BLSTM-2DCNN, methods that apply 2D convolution and 2D max pooling on BLSTM-produced representations to improve text classification across six datasets, achieving state-of-the-art on SST-1 and SST-2.

ABSTRACT

Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks.

Motivation & Objective

• Motivate exploiting both time-step and feature-dimension information in text representations.
• Propose two architectures (BLSTM-2DPooling and BLSTM-2DCNN) that apply 2D pooling and convolution on BLSTM outputs.
• Evaluate on six datasets to demonstrate improvements over state-of-the-art models.

Proposed method

• Transform input text into word embeddings.
• Use BLSTM to capture bidirectional context producing a matrix of features.
• Apply 2D convolution to sample features across both dimensions of the BLSTM output matrix.
• Apply 2D max pooling to obtain a fixed-length text representation.
• Classify with a softmax layer and optimize via cross-entropy with L2 regularization.

Experimental results

Research questions

• RQ1 Do 2D convolution and 2D max pooling on BLSTM outputs improve text classification performance over traditional 1D pooling or CNN approaches?
• RQ2 How do BLSTM-2DPooling and BLSTM-2DCNN perform across tasks like sentiment, subjectivity, and topic classification?
• RQ3 What are the effects of 2D filter sizes and pooling sizes on model performance?

Key findings

• BLSTM-2DCNN achieves highest accuracy on SST-1 and SST-2 datasets (52.4 and 89.5 respectively).
• BLSTM-2DCNN outperforms BLSTM-2DPooling and several baselines across most tasks, including MR, Subj, TREC, and 20Ng.
• BLSTM-2DPooling improves over BLSTM alone but is outperformed by BLSTM-2DCNN and some state-of-the-art models on several datasets.
• BLSTM-2DCNN demonstrates that 2D operations capture dependencies across both time-step and feature dimensions, yielding strong performance on sentence- and document-level classification.
• Sensitivity analyses indicate larger 2D filters can boost performance, e.g., best SST-1 result observed with a 5x5 filter and 5x5 pooling.