[Paper Review] Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding

Rie Johnson, Tong Zhang|arXiv (Cornell University)|Apr 6, 2015

Topic Modeling22 references250 citations

TL;DR

This paper proposes a semi-supervised convolutional neural network for text categorization that learns region embeddings from unlabeled data using a two-view learning framework, improving performance on sentiment and topic classification by directly modeling high-level semantic concepts in small text regions rather than relying on word embeddings. The method achieves state-of-the-art results by leveraging context-aware, concept-based representations that are more compact and effective than traditional word or n-gram embeddings.

ABSTRACT

This paper presents a new semi-supervised framework with convolutional neural networks (CNNs) for text categorization. Unlike the previous approaches that rely on word embeddings, our method learns embeddings of small text regions from unlabeled data for integration into a supervised CNN. The proposed scheme for embedding learning is based on the idea of two-view semi-supervised learning, which is intended to be useful for the task of interest even though the training is done on unlabeled data. Our models achieve better results than previous approaches on sentiment classification and topic classification tasks.

Motivation & Objective

• To address the limitations of word embeddings in capturing high-level semantic concepts relevant to text categorization tasks.
• To improve text categorization performance by learning region embeddings directly from unlabeled data using a two-view semi-supervised framework.
• To develop a representation learning method that combines word similarity and contextual information more effectively than standard n-gram or word embedding approaches.
• To demonstrate that region embeddings trained via context prediction on unlabeled data yield more compact and predictive features than word vector manipulation.
• To show that end-to-end training of CNNs with these region embeddings surpasses previous state-of-the-art results on benchmark datasets.

Proposed method

• Learns tv-embeddings (two-view embeddings) from unlabeled data by training a prediction task on surrounding context, such as presence of task-relevant concepts like 'desire to recommend'.
• Uses a modified CNN architecture that inputs both one-hot vectors and learned tv-embeddings of text regions, enabling joint learning of region-level representations.
• Employs a region embedding transformation (RETEX) that maps text regions to low-dimensional vectors using linear functions of word indicators, allowing efficient representation of complex concepts.
• Constructs region embeddings via a weighted sum of word indicators, where word groups represent semantic similarity and signs encode presence/absence, enabling compact representation of unions of simple concepts.
• Applies ReLU activation to the linear combination of region features, forming a non-linear, low-dimensional feature space that preserves predictive structure.
• Uses labeled data indirectly to define the context prediction task for tv-embedding learning, ensuring the embeddings are tailored to the downstream categorization task.

Experimental results

Research questions

• RQ1 Can region embeddings learned from unlabeled data improve text categorization performance compared to word embeddings?
• RQ2 Does learning embeddings based on context prediction (two-view learning) lead to more effective representations than general-purpose word vector learning?
• RQ3 Can a CNN trained directly on one-hot vectors with region embeddings outperform word-embedding-based CNNs in terms of accuracy and simplicity?
• RQ4 How does the proposed RETEX representation compare to traditional n-gram and word embedding methods in handling data sparsity and capturing semantic concepts?
• RQ5 To what extent do tv-embeddings capture high-level semantic concepts (e.g., sentiment, product recommendation) that individual words cannot?

Key findings

• The proposed method achieves state-of-the-art performance on both sentiment classification and topic classification benchmarks, surpassing previous best results.
• Models using tv-embeddings outperform word-embedding-based CNNs, even when the latter are fine-tuned, due to more compact and task-specific representations.
• The RETEX representation can embed any simple semantic concept (e.g., 'easy to use') into a single dimension, demonstrating superior expressivity over individual n-grams.
• The method reduces data sparsity issues by modeling word similarity and co-occurrence jointly, allowing unions of concepts to be represented with far fewer dimensions than n-grams.
• Theoretical analysis confirms that RETEX can represent any union of simple concepts with a low-dimensional linear transformation, enabling efficient and robust feature learning.
• Empirical results show that tv-embeddings trained on unlabeled data provide more effective input features than post-processing of pre-trained word embeddings, even when the latter are fine-tuned.