[Paper Review] Dynamic Routing Between Capsules

Sara Sabour, Nicholas Frosst+1 more|arXiv (Cornell University)|Oct 26, 2017

Topic Modeling16 references88 citations

TL;DR

Capsule networks (CapsNets) use vectors to represent instantiation parameters and a routing-by-agreement mechanism to dynamically route outputs to higher-level capsules, achieving strong MNIST performance and robustness to overlapping digits.

ABSTRACT

A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.

Motivation & Objective

• Motivate a representation where capsule outputs are vectors encoding instantiation parameters and existence as vector length.
• Introduce dynamic routing by agreement to assign lower-level capsule outputs to appropriate higher-level capsules.
• Show that CapsNets with routing and reconstruction regularization achieve competitive MNIST accuracy with fewer parameters.
• Demonstrate robustness to affine transformations and capability to segment highly overlapping digits.

Proposed method

• Define capsules as groups of neurons with vector outputs where length represents existence probability and orientation encodes instantiation parameters.
• Use transformation matrices to predict higher-level capsule instantiations and routing-by-agreement to adjust coupling coefficients.
• Employ a squashing non-linearity to keep capsule outputs length in [0,1).
• Implement iterative routing (softmax over coupling logits) to refine which lower-level capsules feed which higher-level capsules based on agreement (dot product) between predictions and outputs.
• Train with margin loss per digit class to encourage correct presence/absence signaling; sum across digit capsules.
• Optionally use a reconstruction decoder to regularize pose encoding by reconstructing the input from the target digit capsule.

Experimental results

Research questions

• RQ1 Can a vector-based capsule representation combined with routing-by-agreement outperform CNNs on digit recognition tasks like MNIST?
• RQ2 Does reconstruction as regularization improve the quality of pose encoding and routing performance?
• RQ3 How does CapsNet handle highly overlapping digits and affine transformations compared to traditional CNNs?
• RQ4 What is the impact of routing iterations and model size on classification accuracy and segmentation capability?

Key findings

• CapsNets with routing achieve 0.25% MNIST error with 3 routing iterations and reconstruction regularization.
• CapsNet with routing and no reconstruction achieves 0.34% MNIST error; with routing and reconstruction 0.29%; 1 iteration vs 3 iterations shows performance gains.
• CapsNet on MNIST uses 8.2M parameters (6.8M without reconstruction) versus 35.4M parameters for a comparable CNN baseline.
• CapsNet achieves 0.25% on MNIST and 5.2% on MultiMNIST; reconstruction regularization boosts routing performance.
• CapsNet demonstrates robustness to affine transformations and can segment highly overlapping digits by routing by agreement.
• On CIFAR-10, an ensemble of CapsNets achieves 10.6% test error, comparable to early CNN results; on smallNORB and SVHN, competitive performance is reported.