[Paper Review] Adaptive Routing Between Capsules

Qiang Ren, Shaohua Shang+1 more|arXiv (Cornell University)|Nov 19, 2019

Advanced Neural Network Applications17 references4 citations

TL;DR

This paper proposes an adaptive routing algorithm for capsule networks that eliminates the coupling coefficient to prevent gradient vanishing during deep stacking, enabling stable training of deeper architectures. By replacing iterative routing with a learnable hyper-parameter λ, the method reduces computation while achieving state-of-the-art performance on MNIST, Fashion-MNIST, SVHN, and CIFAR-10.

ABSTRACT

Capsule network is the most recent exciting advancement in the deep learning field and represents positional information by stacking features into vectors. The dynamic routing algorithm is used in the capsule network, however, there are some disadvantages such as the inability to stack multiple layers and a large amount of computation. In this paper, we propose an adaptive routing algorithm that can solve the problems mentioned above. First, the low-layer capsules adaptively adjust their direction and length in the routing algorithm and removing the influence of the coupling coefficient on the gradient propagation, so that the network can work when stacked in multiple layers. Then, the iterative process of routing is simplified to reduce the amount of computation and we introduce the gradient coefficient $λ$. Further, we tested the performance of our proposed adaptive routing algorithm on CIFAR10, Fashion-MNIST, SVHN and MNIST, while achieving better results than the dynamic routing algorithm.

Motivation & Objective

• Address the gradient vanishing problem in capsule networks when stacking multiple layers, which hinders deep network training.
• Overcome the limitations of dynamic routing, particularly the sparsity of coupling coefficients that suppress gradient flow.
• Reduce computational cost by simplifying the iterative routing process without sacrificing performance.
• Enable stable training of deeper capsule networks by decoupling gradient propagation from coupling coefficients.
• Optimize performance through a learnable hyper-parameter λ that replaces the number of routing iterations.

Proposed method

• Replace the dynamic routing mechanism’s coupling coefficient c_ij with adaptive updates to low-level capsule states, eliminating c_ij’s influence on gradient flow.
• Update only the low-level capsule activations and routing vectors at each iteration, improving gradient transmission to earlier layers.
• Introduce a hyper-parameter λ that controls the routing strength, replacing the fixed number of iterations in dynamic routing.
• Simplify the routing process by removing iterative coupling coefficient updates, reducing computational overhead.
• Use λ to modulate gradient amplification, enabling more effective backpropagation through deep capsule architectures.
• Design the routing process so that low-level capsules adaptively align with high-level capsule predictions, improving feature representation.

Experimental results

Research questions

• RQ1 Why does dynamic routing cause gradient vanishing in deep capsule networks, and how does the coupling coefficient c_ij contribute to this issue?
• RQ2 Can the gradient vanishing problem in stacked capsule networks be resolved by removing the coupling coefficient from the routing process?
• RQ3 How does the proposed adaptive routing algorithm improve performance and stability in deeper capsule network architectures?
• RQ4 To what extent can the iterative routing process be simplified using a single hyper-parameter λ without degrading model accuracy?
• RQ5 What is the optimal value of λ for achieving high performance across different datasets and network depths?

Key findings

• The adaptive routing algorithm achieves state-of-the-art performance on MNIST, outperforming the original dynamic routing method.
• On Fashion-MNIST, the model reaches 93.71% accuracy with 4 capsule layers and λ = 2, demonstrating improved performance over dynamic routing.
• On CIFAR-10, the model achieves 78.68% accuracy with 4 layers and λ = 3, showing consistent improvement over baseline dynamic routing.
• When λ is set too low (e.g., 0.0001 or below), the network collapses due to vanishing gradients, especially in deeper architectures.
• The performance is sensitive to λ: optimal values are λ = 2 or 3 on Fashion-MNIST and λ = 1 or 3 on CIFAR-10, indicating a trade-off between gradient amplification and noise.
• The adaptive routing method enables stable training of up to 4 capsule layers, which is not feasible with standard dynamic routing due to gradient suppression.