[Paper Review] Capsule Network is Not More Robust than Convolutional Network

Jindong Gu, Volker Tresp+1 more|arXiv (Cornell University)|Mar 29, 2021

Adversarial Robustness in Machine Learning16 references4 citations

TL;DR

This paper challenges the widely held belief that Capsule Networks (CapsNets) are more robust than Convolutional Neural Networks (ConvNets). Through comprehensive ablation studies, it identifies that dynamic routing and transformation matrices in CapsNets actually harm robustness to affine transformations and overlapping digits, while components like squashing and class-conditional reconstruction improve semantic compactness. The authors enhance standard ConvNets with these beneficial components, achieving superior robustness compared to CapsNets across all evaluated tasks.

ABSTRACT

The Capsule Network is widely believed to be more robust than Convolutional Networks. However, there are no comprehensive comparisons between these two networks, and it is also unknown which components in the CapsNet affect its robustness. In this paper, we first carefully examine the special designs in CapsNet that differ from that of a ConvNet commonly used for image classification. The examination reveals five major new/different components in CapsNet: a transformation process, a dynamic routing layer, a squashing function, a marginal loss other than cross-entropy loss, and an additional class-conditional reconstruction loss for regularization. Along with these major differences, we conduct comprehensive ablation studies on three kinds of robustness, including affine transformation, overlapping digits, and semantic representation. The study reveals that some designs, which are thought critical to CapsNet, actually can harm its robustness, i.e., the dynamic routing layer and the transformation process, while others are beneficial for the robustness. Based on these findings, we propose enhanced ConvNets simply by introducing the essential components behind the CapsNet's success. The proposed simple ConvNets can achieve better robustness than the CapsNet.

Motivation & Objective

• To challenge the prevailing assumption that Capsule Networks are inherently more robust than Convolutional Neural Networks.
• To identify which components in CapsNet genuinely contribute to robustness versus those that degrade performance.
• To evaluate the impact of each CapsNet-specific component—dynamic routing, transformation matrices, squashing, marginal loss, and reconstruction—on three robustness metrics: affine transformation, overlapping digits, and semantic representation.
• To demonstrate that simple ConvNets enhanced with key CapsNet components can outperform CapsNet in robustness.

Proposed method

• Conduct ablation studies on CapsNet components using three robustness benchmarks: affine transformations, overlapping digit recognition, and semantic representation compactness.
• Implement a modified ConvNet baseline with global average pooling, which improves spatial invariance and generalization.
• Introduce and evaluate key CapsNet components in isolation: dynamic routing, transformation matrices, squashing function, marginal loss, and class-conditional reconstruction.
• Use a compactness score based on KL divergence between normalized variance of representation vectors and a uniform prior to quantify semantic representation quality.
• Apply perturbations to capsule and feature activations to analyze semantic sensitivity in reconstructed images.
• Train and compare models on MNIST and FMNIST datasets under controlled settings to isolate component effects.

Experimental results

Research questions

• RQ1 Does the dynamic routing mechanism in CapsNets truly improve robustness to affine transformations, or does it degrade performance?
• RQ2 To what extent do transformation matrices and non-shared weights in CapsNets contribute to recognizing overlapping digits?
• RQ3 Can the squashing function and class-conditional reconstruction loss improve semantic representation compactness in standard ConvNets?
• RQ4 Is the perceived robustness of CapsNets due to its architecture or simply because of a weak ConvNet baseline used in prior comparisons?
• RQ5 Can a simple ConvNet enhanced with key CapsNet components outperform CapsNet in all robustness metrics?

Key findings

• The standard ConvNet baseline used in prior CapsNet comparisons lacks global average pooling, weakening its spatial invariance and generalization, which undermines fair comparison.
• Dynamic routing in CapsNets harms robustness to affine transformations, contradicting the common belief that it enhances invariance.
• The ability of CapsNets to recognize overlapping digits is primarily due to the increased modeling capacity from non-shared transformation matrices, not routing or vector representations.
• The squashing function and class-conditional reconstruction loss significantly improve semantic representation compactness, and these components can be effectively transferred to ConvNets.
• Enhanced ConvNets incorporating squashing and class-conditional reconstruction outperform CapsNet in all three robustness benchmarks, including affine transformation, overlapping digits, and compactness.
• The compactness score for ConvNet-CR-SF (with reconstruction and squashing) reaches 0.3192 for trans-Y, significantly higher than CapsNet’s 0.0464, demonstrating superior semantic representation.