[Paper Review] Dense and Diverse Capsule Networks: Making the Capsules Learn Better

Sai Samarth R Phaye, Apoorva Sikka+2 more|arXiv (Cornell University)|May 10, 2018

Advanced Neural Network Applications14 references45 citations

TL;DR

The paper introduces Dense Capsule Networks (DCNet) and Diverse Capsule Networks (DCNet++) that replace standard convolutions with densely connected convolutions to learn richer primary capsules, achieving faster convergence and improved accuracy on MNIST and other datasets, including strong CIFAR-10 results with reduced parameters.

ABSTRACT

Past few years have witnessed exponential growth of interest in deep learning methodologies with rapidly improving accuracies and reduced computational complexity. In particular, architectures using Convolutional Neural Networks (CNNs) have produced state-of-the-art performances for image classification and object recognition tasks. Recently, Capsule Networks (CapsNet) achieved significant increase in performance by addressing an inherent limitation of CNNs in encoding pose and deformation. Inspired by such advancement, we asked ourselves, can we do better? We propose Dense Capsule Networks (DCNet) and Diverse Capsule Networks (DCNet++). The two proposed frameworks customize the CapsNet by replacing the standard convolutional layers with densely connected convolutions. This helps in incorporating feature maps learned by different layers in forming the primary capsules. DCNet, essentially adds a deeper convolution network, which leads to learning of discriminative feature maps. Additionally, DCNet++ uses a hierarchical architecture to learn capsules that represent spatial information in a fine-to-coarser manner, which makes it more efficient for learning complex data. Experiments on image classification task using benchmark datasets demonstrate the efficacy of the proposed architectures. DCNet achieves state-of-the-art performance (99.75%) on MNIST dataset with twenty fold decrease in total training iterations, over the conventional CapsNet. Furthermore, DCNet++ performs better than CapsNet on SVHN dataset (96.90%), and outperforms the ensemble of seven CapsNet models on CIFAR-10 by 0.31% with seven fold decrease in number of parameters.

Motivation & Objective

• Improve CapsNet performance and convergence through densely connected convolutions.
• Leverage dense feature concatenation to learn diversified primary capsules.
• Address limitations on complex datasets (e.g., CIFAR-10) with hierarchical, coarse-to-fine capsule representations.
• Reduce model complexity while maintaining or improving accuracy.
• Explore reconstruction decoder enhancements inspired by DenseNet ideas.

Proposed method

• Replace standard Conv layers in CapsNet with an 8-layer dense convolutional subnetwork that concatenates features across layers to form primary capsules.
• Form 32 eight-dimensional primary capsules at the final dense block, then apply routing-by-agreement to produce DigitCaps.
• Modify the decoder to concatenate first-layer and second-layer features for better reconstructions.
• Introduce DCNet++ as a hierarchical, multi-level architecture (three-level) that learns fine-to-coarse spatial features and shares representations across levels.
• Use guided back-propagation to analyze activations and motivate diverse capsule representations across scales.
• Train with same parameter counts as baselines for fair comparison and adjust reconstruction loss scaling to balance training.

Experimental results

Research questions

• RQ1 Can dense, skip-connected convolutions improve the quality of capsules and speed up convergence compared to baseline CapsNet?
• RQ2 Does introducing hierarchy and diverse primary capsules (DCNet++) improve performance on complex datasets like CIFAR-10 relative to single CapsNet and DCNet?
• RQ3 What is the impact of replacing pooling with dense connections on retaining spatial information in capsules?
• RQ4 Can reconstruction decoders benefit from dense, multi-layer fusion to yield better reconstructions and assist learning?

Key findings

• DCNet achieves 99.75% test accuracy on MNIST in 50 epochs, with twenty-fold fewer total training iterations than conventional CapsNet.
• DCNet attains 94.64% on Fashion-MNIST (100 epochs), outperforming the baseline CapsNet under comparable settings.
• On SVHN, DCNet improves to 95.59% over a replicated CapsNet (93.23%), and DCNet++ further improves to 96.90%.
• DCNet++ on CIFAR-10 reaches 89.71% accuracy with 13.4M parameters, beating the seven-model CapsNet ensemble (89.40%) with significantly fewer parameters.
• DCNet shows notable improvements on SmallNORB and brain-tumor datasets, and DCNet++ provides a hierarchical, fine-to-coarse learning mechanism to tackle complex data.
• Overall, DCNet variants achieve competitive or superior performance with reduced or comparable parameter counts relative to existing CapsNet baselines and ensembles.