[Paper Review] Sparse Unsupervised Capsules Generalize Better

David Rawlinson, Abdelrahman Ahmed+1 more|arXiv (Cornell University)|Apr 17, 2018

Imbalanced Data Classification TechniquesComputer Science23 references47 citations

TL;DR

The paper shows that removing supervised masking in unsupervised capsule training degrades capsule properties; introducing sparse unsupervised training restores these properties and improves generalization to affNIST, achieving up to 90.12% with an SVM classifier.

ABSTRACT

We show that unsupervised training of latent capsule layers using only the reconstruction loss, without masking to select the correct output class, causes a loss of equivariances and other desirable capsule qualities. This implies that supervised capsules networks can't be very deep. Unsupervised sparsening of latent capsule layer activity both restores these qualities and appears to generalize better than supervised masking, while potentially enabling deeper capsules networks. We train a sparse, unsupervised capsules network of similar geometry to Sabour et al (2017) on MNIST, and then test classification accuracy on affNIST using an SVM layer. Accuracy is improved from benchmark 79% to 90%.

Motivation & Objective

• Investigate how unsupervised training of latent capsule layers affects capsule qualities and generalization.
• Determine whether sparsity can restore capsule-specific equivariances without supervised masking.
• Compare sparse unsupervised capsule networks to supervised masking approaches on MNIST/affNIST.
• Assess generalization performance of sparse unsupervised capsules with an external classifier.

Proposed method

• Modify SUPCAPS architecture to remove masking and margin loss and observe degeneration.
• Introduce a sparse training scheme where only top-k latent capsules (by routing support) contribute to output.
• Apply an online lifetime sparsity constraint to ensure all capsules participate over time.
• Compute per-batch and per-image capsule activations to enforce sparsity via a ranking-based mask m_jk.
• Train on MNIST and evaluate generalization to affNIST using an SVM classifier on capsule outputs.
• Provide reconstruction and routing analyses to compare with dense unsupervised and supervised-capacity baselines.

Experimental results

Research questions

• RQ1 Can unsupervised latent capsule training without masking preserve capsule-specific equivariances and routing dynamics?
• RQ2 Does sparsifying latent capsule activity restore desirable capsule properties and improve generalization to affine-transformed data?
• RQ3 How does sparse unsupervised SPARSECAPS compare to SUPCAPS and conventional CNNs on MNIST/affNIST?
• RQ4 What is the effect of lifetime sparsity constraints on capsule utilization and performance?

Key findings

• Dense unsupervised capsules without sparsity lose equivariances and routing selectivity, performing poorly on affNIST.
• Sparse unsupervised training reestablishes capsule specialization and dynamic routing, improving generalization.
• SPARSECAPS with an SVM classifier achieves 90.12% on affNIST when trained on MNIST (vs. 79% for SUPCAPS).
• Unmasked dense unsupervised capsules perform poorly (affNIST around 60.63% or 15.70% with different setups).
• K-sparse style masking results in approximately 1 fully active latent capsule with a few partially active capsules, balancing capacity and specialization.
• Equivariances observed in sparse latent capsules include variations in stroke width, skew, and form morphing, indicating meaningful latent factors.