[论文解读] Capsule Network Performance on Complex Data

Edgar Xi, Selina Bing+1 more|arXiv (Cornell University)|Dec 10, 2017

Advanced Neural Network Applications参考文献 2被引用 180

一句话总结

本文评估胶囊网络在 CIFAR-10 上的表现，探讨通过修改（集成、卷积层、重建缩放）来提升超越 MNIST 结果的准确性，并讨论在复杂数据上的局限性。

ABSTRACT

In recent years, convolutional neural networks (CNN) have played an important role in the field of deep learning. Variants of CNN's have proven to be very successful in classification tasks across different domains. However, there are two big drawbacks to CNN's: their failure to take into account of important spatial hierarchies between features, and their lack of rotational invariance. As long as certain key features of an object are present in the test data, CNN's classify the test data as the object, disregarding features' relative spatial orientation to each other. This causes false positives. The lack of rotational invariance in CNN's would cause the network to incorrectly assign the object another label, causing false negatives. To address this concern, Hinton et al. propose a novel type of neural network using the concept of capsules in a recent paper. With the use of dynamic routing and reconstruction regularization, the capsule network model would be both rotation invariant and spatially aware. The capsule network has shown its potential by achieving a state-of-the-art result of 0.25% test error on MNIST without data augmentation such as rotation and scaling, better than the previous baseline of 0.39%. To further test out the application of capsule networks on data with higher dimensionality, we attempt to find the best set of configurations that yield the optimal test error on CIFAR10 dataset.

研究动机与目标

• 评估胶囊网络在超越 MNIST 的高维数据（CIFAR-10）中的适用性。
• 研究用于改进复杂数据性能的体系结构与训练修改。
• 分析重建正则化及其对学习更丰富表征的影响。
• 将胶囊网络的变体与基线 MNIST 配置进行比较并讨论局限性。

提出的方法

• 以 Hinton 的基于 MNIST 的胶囊网络，包含 3 个色彩通道作为基线。
• 尝试堆叠更多的胶囊层并增加初级胶囊的数量。
• 在测试时使用集成平均来组合多个模型。
• 调整重建损失缩放和胶囊层之前的卷积层数量。
• 测试替代 squash 函数的定制激活函数。
• 包括一个“上述均不适用”类别以评估对准确性的影响。

实验结果

研究问题

• RQ1 在适当的体系结构变更下，胶囊网络能否在 CIFAR-10 上实现有竞争力的准确率？
• RQ2 增加卷积层、更多胶囊以及集成对 CIFAR-10 验证准确率的影响是什么？
• RQ3 重建缩放如何影响在高维数据上的过拟合与收敛？
• RQ4 定制激活函数或额外的胶囊层会提升还是降低在 CIFAR-10 上的性能？

主要发现

模型	25 轮验证准确率	50 轮验证准确率
MNIST Model Baseline	67.51%	68.93%
64 Capsule Layers	60.54%	64.67%
4-Model Ensemble (4 Ensemble)	68.97%	70.78%
2-Convolution Layers (2 Conv)	68.14%	69.34%
4 Ensemble + 2 Conv	70.34%	71.50%
7 Ensemble + 2 Conv	70.50%	______
4 Ensemble + 2 Conv + 0.0001 Reconstruction Scaling	69.21%	______
Stack Additional Capsule Layer	10.11%	______

• 最佳模型：4 集成，含 2 层卷积，在 50 轮时达到 71.550% 的验证准确率。
• 添加一个卷积层使验证准确率提高 0.41%。
• 4 模型集成在 50 轮时比基线提高验证准确率 1.85%。
• 7 模型集成并额外的卷积在相对于 4-ensemble+2-conv 的提升方面收效甚微，但由于资源限制未完成测试。
• 堆叠额外的胶囊层的表现比基线差得多。
• 重建缩放和增加胶囊类型数量的表现均低于预期。