[論文レビュー] Capsule Network Performance on Complex Data

Edgar Xi, Selina Bing+1 more|arXiv (Cornell University)|Dec 10, 2017

Advanced Neural Network Applications参考文献 2被引用数 180

ひとこと要約

この論文は capsule ネットワークが CIFAR-10 でどのように機能するかを評価し、MNIST の結果を超える精度を目指すためのエン ensemble、畳み込み層の追加、再構成スケーリングといった修正を検討し、複雑なデータに対する制約について論じている。

ABSTRACT

In recent years, convolutional neural networks (CNN) have played an important role in the field of deep learning. Variants of CNN's have proven to be very successful in classification tasks across different domains. However, there are two big drawbacks to CNN's: their failure to take into account of important spatial hierarchies between features, and their lack of rotational invariance. As long as certain key features of an object are present in the test data, CNN's classify the test data as the object, disregarding features' relative spatial orientation to each other. This causes false positives. The lack of rotational invariance in CNN's would cause the network to incorrectly assign the object another label, causing false negatives. To address this concern, Hinton et al. propose a novel type of neural network using the concept of capsules in a recent paper. With the use of dynamic routing and reconstruction regularization, the capsule network model would be both rotation invariant and spatially aware. The capsule network has shown its potential by achieving a state-of-the-art result of 0.25% test error on MNIST without data augmentation such as rotation and scaling, better than the previous baseline of 0.39%. To further test out the application of capsule networks on data with higher dimensionality, we attempt to find the best set of configurations that yield the optimal test error on CIFAR10 dataset.

研究の動機と目的

• Assess the applicability of capsule networks to higher-dimensional data beyond MNIST (CIFAR-10).
• Investigate architectural and training modifications to improve performance on complex data.
• Analyze reconstruction regularization and its impact on learning richer representations.
• Compare capsule network variants to baseline MNIST configurations and discuss limitations.

提案手法

• Start from Hinton’s MNIST-based capsule network with 3 color channels as a baseline.
• Experiment with stacking more capsule layers and increasing the number of primary capsules.
• Use ensemble averaging to combine multiple models at test time.
• Tune reconstruction loss scaling and the number of convolution layers before the capsule layer.
• Test a customized activation function in place of the squash function.
• Include a 'none of the above' category to assess impact on accuracy.

実験結果

リサーチクエスチョン

• RQ1 Can capsule networks achieve competitive CIFAR-10 accuracy with appropriate architectural changes?
• RQ2 What is the impact of adding convolution layers, more capsules, and ensembling on validation accuracy for CIFAR-10?
• RQ3 How does reconstruction scaling affect overfitting and convergence on higher-dimensional data?
• RQ4 Does a customized activation or additional capsule layer improve or degrade performance on CIFAR-10?

主な発見

Model	Validation Accuracy 25 Epochs	Validation Accuracy 50 Epochs
MNIST Model Baseline	67.51%	68.93%
64 Capsule Layers	60.54%	64.67%
4-Model Ensemble (4 Ensemble)	68.97%	70.78%
2-Convolution Layers (2 Conv)	68.14%	69.34%
4 Ensemble + 2 Conv	70.34%	71.50%
7 Ensemble + 2 Conv	70.50%	______
4 Ensemble + 2 Conv + 0.0001 Reconstruction Scaling	69.21%	______
Stack Additional Capsule Layer	10.11%	______

• Best model: 4-ensemble with 2 convolution layers achieving 71.550% validation accuracy at 50 epochs.
• Adding a convolution layer increases validation accuracy by 0.41%.
• A 4-model ensemble increases validation accuracy by 1.85% over the baseline at 50 epochs.
• 7-model ensemble with an extra convolution yielded marginal gains over 4-ensemble+2-conv, but not tested to completion due to resources.
• Stacking an additional capsule layer performed drastically worse than the baseline.
• Reconstruction scaling and increasing the number of capsule types underperformed relative to expectations.