[Paper Review] Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

Rie Johnson, Tong Zhang|arXiv (Cornell University)|Dec 1, 2014

Topic Modeling24 references198 citations

TL;DR

This paper proposes a novel convolutional neural network (CNN) approach for text categorization that directly processes high-dimensional one-hot word vectors instead of pre-trained word embeddings, enabling effective learning of local text region representations while preserving word order. The method achieves state-of-the-art performance on both sentiment and topic classification by capturing high-order n-grams—even those unseen in training—through end-to-end learning, outperforming traditional bag-of-n-gram and word-embedding-based models.

ABSTRACT

Convolutional neural network (CNN) is a neural network that can make use of the internal structure of data such as the 2D structure of image data. This paper studies CNN on text categorization to exploit the 1D structure (namely, word order) of text data for accurate prediction. Instead of using low-dimensional word vectors as input as is often done, we directly apply CNN to high-dimensional text data, which leads to directly learning embedding of small text regions for use in classification. In addition to a straightforward adaptation of CNN from image to text, a simple but new variation which employs bag-of-word conversion in the convolution layer is proposed. An extension to combine multiple convolution layers is also explored for higher accuracy. The experiments demonstrate the effectiveness of our approach in comparison with state-of-the-art methods.

Motivation & Objective

• To address the limitation of bag-of-words and n-gram models in preserving word order, especially in sentiment classification.
• To investigate whether direct application of CNNs to high-dimensional sparse text data can outperform word embedding-based approaches in text categorization.
• To explore the effectiveness of CNNs in capturing high-order n-grams (e.g., tri-grams) that are not explicitly present in training data.
• To develop a simple yet effective CNN architecture that avoids reliance on pre-trained word vectors while maintaining fast training and high accuracy.
• To demonstrate that CNNs can generalize to unseen n-grams by learning contextual representations through end-to-end training.

Proposed method

• The proposed method applies 1D convolutional layers directly to high-dimensional one-hot encoded word vectors, bypassing word embedding lookup.
• Two variants are introduced: seq-CNN, a direct adaptation of image CNNs to text, and bow-CNN, which applies a bag-of-words transformation within the convolution layer.
• The model uses max-pooling over the output of convolutional filters to produce fixed-length feature vectors for classification.
• A multi-layer CNN architecture is explored, combining multiple convolutional layers to learn hierarchical representations of text regions.
• Efficient GPU-based computation is used to handle the sparsity of one-hot vectors, making training feasible despite high dimensionality.
• The network is trained end-to-end in a supervised manner, with word vectors learned as part of the training process.

Experimental results

Research questions

• RQ1 Can a CNN model trained directly on one-hot word vectors outperform traditional bag-of-n-gram and word-embedding-based models in text categorization?
• RQ2 To what extent can CNNs learn and generalize to high-order n-grams (e.g., tri-grams) that were not present in the training data?
• RQ3 How does preserving word order through 1D convolution improve performance on sentiment and topic classification tasks?
• RQ4 Does the proposed approach maintain fast training and prediction speed while achieving higher accuracy than existing CNN-based text models?
• RQ5 Is the combination of multiple convolutional layers effective in capturing complementary patterns across different text region sizes?

Key findings

• The seq-CNN variant outperforms bow-CNN on sentiment classification tasks, achieving state-of-the-art results on the IMDB and Rotten Tomatoes datasets.
• The bow-CNN variant performs better than seq-CNN on topic classification, demonstrating the effectiveness of bag-of-words transformation in capturing topic-relevant phrases.
• The model successfully generalizes to unseen tri-grams such as 'am entirely satisfied' and 'best concept ever', which are not explicitly present in the training data but contribute to accurate prediction.
• The system achieves higher accuracy than conventional bag-of-n-gram models and more complex prior CNN models, even without pre-trained word embeddings.
• The use of multiple convolutional layers leads to further performance improvements by combining different types of text region embeddings.
• Empirical analysis confirms that the learned embeddings reflect semantic proximity to target classes, with similar embedded vectors corresponding to text regions with similar sentiment impact.