[Paper Review] Capsule Networks with Max-Min Normalization

Zhao Zhen, Ashley Kleinhans+3 more|arXiv (Cornell University)|Mar 22, 2019

Advanced Neural Network Applications15 references40 citations

TL;DR

The paper replaces Softmax routing in CapsNets with a scale-invariant Max-Min normalization, improving accuracy across five datasets and enabling more routing iterations without performance loss.

ABSTRACT

Capsule Networks (CapsNet) use the Softmax function to convert the logits of the routing coefficients into a set of normalized values that signify the assignment probabilities between capsules in adjacent layers. We show that the use of Softmax prevents capsule layers from forming optimal couplings between lower and higher-level capsules. Softmax constrains the dynamic range of the routing coefficients and leads to probabilities that remain mostly uniform after several routing iterations. Instead, we propose the use of Max-Min normalization. Max-Min performs a scale-invariant normalization of the logits that allows each lower-level capsule to take on an independent value, constrained only by the bounds of normalization. Max-Min provides consistent improvement in test accuracy across five datasets and allows more routing iterations without a decrease in network performance. A single CapsNet trained using Max-Min achieves an improved test error of 0.20% on the MNIST dataset. With a simple 3-model majority vote, we achieve a test error of 0.17% on MNIST.

Motivation & Objective

• Motivate improvements to CapsNet dynamic routing by addressing Softmax limitations.
• Propose Max-Min normalization as a scale-invariant routing coefficient normalization.
• Evaluate Max-Min across multiple datasets and routing iterations.
• Demonstrate improved MNIST performance and potential ensemble gains.

Proposed method

• Replace Softmax with Max-Min in the routing procedure for CapsNets.
• Allow routing coefficients to be independently bounded between 0 and 1.
• Use a three-layer CapsNet architecture with PrimaryCaps and DigitCaps layers.
• Train with standard margin and reconstruction losses, plus routing iterations.
• Compare evolution of logits and routing coefficients under Softmax vs. Max-Min.

Experimental results

Research questions

• RQ1 Does Max-Min normalization improve CapsNet routing quality over Softmax across datasets?
• RQ2 How does Max-Min affect routing coefficient dynamics and feature binding between capsule layers?
• RQ3 Can increasing routing iterations with Max-Min boost accuracy without overfitting?
• RQ4 What is the impact of Max-Min on MNIST performance relative to Softmax?
• RQ5 Are there qualitative differences in DigitCaps outputs under the two normalizations?

Key findings

• Max-Min consistently improves test accuracy across MNIST, rMNIST, fMNIST, bMNIST, and CIFAR10 compared to Softmax.
• Max-Min allows more routing iterations without degrading performance and reduces overfitting.
• Single CapsNet with Max-Min achieves 0.20% test error on MNIST; a 3-model ensemble reaches 0.17%.
• Routing coefficients under Max-Min show larger variation and enable multiple higher-level capsules to receive high assignments.
• MNIST results with Max-Min closely approach state-of-the-art while using minimal augmentation.