[Paper Review] Text Classification using Capsules

Jaeyoung Kim, Sion Jang+2 more|arXiv (Cornell University)|Aug 12, 2018

Text and Document Classification TechnologiesComputer Science29 references65 citations

TL;DR

The paper investigates applying capsule networks to text classification, introducing a static routing variant with an ELU-gate and showing competitive results across seven benchmarks compared to CNN baselines.

ABSTRACT

This paper presents an empirical exploration of the use of capsule networks for text classification. While it has been shown that capsule networks are effective for image classification, their validity in the domain of text has not been explored. In this paper, we show that capsule networks indeed have the potential for text classification and that they have several advantages over convolutional neural networks. We further suggest a simple routing method that effectively reduces the computational complexity of dynamic routing. We utilized seven benchmark datasets to demonstrate that capsule networks, along with the proposed routing method provide comparable results.

Motivation & Objective

• Demonstrate that capsule networks can perform text classification effectively.
• Introduce a static routing variant as an alternative to dynamic routing for text data.
• Propose ELU-gate as a mechanism to propagate relevant information without losing spatial context.
• Evaluate performance on seven standard text classification datasets against CNN baselines.

Proposed method

• Apply capsule networks to text by mapping documents to vectors via convolutional capsules and a final text capsule layer.
• Use a gating mechanism (ELU-gate) to select activated features without relying on pooling.
• Implement two routing schemes between convolutional capsules and text capsules: dynamic routing (as in Sabour et al.) and a proposed static routing.
• Increase the kernel size in the convolutional capsule layer to enlarge the receptive field and avoid pooling.
• Train with preprocessed GloVe word vectors and compare to CNN baselines; report accuracies on seven datasets.

Experimental results

Research questions

• RQ1 Can capsule networks achieve competitive text classification accuracy compared to CNNs on standard benchmarks?
• RQ2 Does static routing offer advantages over dynamic routing in text classification in terms of accuracy and computational efficiency?
• RQ3 What is the impact of architectural choices (ELU-gate, large kernel conv capsules) on performance for text data?
• RQ4 How do capsule-based models behave across diverse datasets (e.g., news, sentiment, Q&A)?

Key findings

• Capsule networks with both dynamic and static routing achieve competitive accuracies across seven datasets compared to CNN baselines.
• Static routing generally yields higher accuracy than dynamic routing on the evaluated datasets.
• Static routing reduces computational complexity while maintaining or improving performance in text classification.
• ELU-gate ablations show improved accuracy over variants without gating, indicating benefits from information-preserving feature selection.
• On 20news, Reuters10, MR(2004), MR(2005), TREC-QA, MPQA, and IMDb, CapsNet-static-routing achieves accuracies of 87.17, 87.52, 89.6, 80.98, 94.84, 90.57, 89.72 respectively; CapsNet-dynamic-routing achieves 86.45, 86.72, 88.1, 81.00, 93.80, 89.60, 89.80; CNN-non-static* achieves 86.6, 87.4, 88.0, 81.3, 92.7, 89.9, 90.36.