[Paper Review] Investigating Capsule Networks with Dynamic Routing for Text Classification

Wei Zhao, Jianbo Ye+4 more|arXiv (Cornell University)|Mar 29, 2018

Text and Document Classification TechnologiesComputer Science22 references101 citations

TL;DR

This paper proposes capsule networks with three routing-stabilization strategies for text classification and demonstrates competitive results on six benchmarks, with notable gains in single-label to multi-label transfer tasks.

ABSTRACT

In this study, we explore capsule networks with dynamic routing for text classification. We propose three strategies to stabilize the dynamic routing process to alleviate the disturbance of some noise capsules which may contain "background" information or have not been successfully trained. A series of experiments are conducted with capsule networks on six text classification benchmarks. Capsule networks achieve state of the art on 4 out of 6 datasets, which shows the effectiveness of capsule networks for text classification. We additionally show that capsule networks exhibit significant improvement when transfer single-label to multi-label text classification over strong baseline methods. To the best of our knowledge, this is the first work that capsule networks have been empirically investigated for text modeling.

Motivation & Objective

• Motivate the use of capsule networks to model text as part-whole relationships and preserve instantiation parameters beyond traditional CNN/RNN representations.
• Develop stabilization strategies for dynamic routing to reduce noise from background words.
• Evaluate Capsule-A and Capsule-B architectures on six text classification benchmarks with pre-trained word embeddings.
• Demonstrate robustness and transfer capability in single-label to multi-label text classification, especially on Reuters datasets.

Proposed method

• Use an n-gram convolutional layer to extract local text features.
• Replace scalar detectors with vector capsules in a four-layer architecture (primary, convolutional capsule, fully connected capsule).
• Employ dynamic routing with three stabilization strategies: orphan category, leaky-softmax, and coefficients amendment.
• Experiment with two capsule architectures (Capsule-A and Capsule-B) and compare to strong baselines across six datasets.
• Train with pre-trained Word2Vec embeddings, using Adam optimization and three routing iterations.

Experimental results

Research questions

• RQ1 Can capsule networks with dynamic routing improve text classification performance over traditional CNN/LSTM baselines?
• RQ2 Do stabilization strategies (orphan category, leaky-softmax, coefficient amendment) reduce noise from background words in text data?
• RQ3 Is Capsule-B superior to Capsule-A due to its multi-gram input configuration?
• RQ4 How well do capsule networks transfer from single-label to multi-label text classification tasks?

Key findings

• Capsule networks achieve best results on 4 of 6 benchmarks, showing competitive performance across diverse text classification tasks.
• Capsule-B consistently outperforms Capsule-A, likely due to using multiple n-gram inputs (3,4,5) to capture richer features.
• Three routing-stabilization strategies collectively improve routing robustness against noisy capsules.
• In multi-label transfer on Reuters datasets, capsule networks show substantial improvements over strong baselines, especially when only single-label data is available for training.
• Routing iterations (3) lead to faster convergence and lower end losses compared to fewer iterations.
• Ablation studies confirm that removing any stabilization component degrades performance.