[论文解读] The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning

Aditya Sharma, Nikolas Wolfe+1 more|arXiv (Cornell University)|Jan 16, 2017

Neural Networks and Applications参考文献 14被引用 13

一句话总结

本文提出对训练好的神经网络中的整个神经元进行剪枝，而非单个权重，证明在不微调的情况下，串行剪枝40–70%的神经元仅导致极小的精度下降。该研究挑战了剪枝领域长期存在的假设，表明神经元重要性分布不均，且神经元之间的相互依赖性使得传统方法所依赖的线性或二次误差近似失效。

ABSTRACT

How much can pruning algorithms teach us about the fundamentals of learning representations in neural networks? And how much can these fundamentals help while devising new pruning techniques? A lot, it turns out. Neural network pruning has become a topic of great interest in recent years, and many different techniques have been proposed to address this problem. The decision of what to prune and when to prune necessarily forces us to confront our assumptions about how neural networks actually learn to represent patterns in data. In this work, we set out to test several long-held hypotheses about neural network learning representations, approaches to pruning and the relevance of one in the context of the other. To accomplish this, we argue in favor of pruning whole neurons as opposed to the traditional method of pruning weights from optimally trained networks. We first review the historical literature, point out some common assumptions it makes, and propose methods to demonstrate the inherent flaws in these assumptions. We then propose our novel approach to pruning and set about analyzing the quality of the decisions it makes. Our analysis led us to question the validity of many widely-held assumptions behind pruning algorithms and the trade-offs we often make in the interest of reducing computational complexity. We discovered that there is a straightforward way, however expensive, to serially prune 40-70% of the neurons in a trained network with minimal effect on the learning representation and without any re-training. It is to be noted here that the motivation behind this work is not to propose an algorithm that would outperform all existing methods, but to shed light on what some inherent flaws in these methods can teach us about learning representations and how this can lead us to superior pruning techniques.

研究动机与目标

• 挑战广泛持有的假设，即剪枝应针对单个权重而非整个神经元。
• 检验关于误差曲面近似和神经元独立性的常见剪枝假设的有效性。
• 通过实证研究证明，尽管计算成本较高，剪枝整个神经元仍可实现高倍率压缩且性能下降极小。
• 揭示相关神经元与无关神经元之间的隐藏相互依赖关系，表明剪枝会影响网络动力学的整体结构。
• 倡导剪枝范式的转变，强调应基于对学习表征的理解，而非基于启发式权重移除。

提出的方法

• 提出一种暴力、串行的神经元剪枝方法，通过单独评估每个神经元移除对网络输出的影响来衡量其重要性。
• 在每次移除神经元后对网络性能进行完整评估，避免依赖基于梯度或海森矩阵的近似方法。
• 利用回归问题和误差曲面可视化分析神经元移除周围的真实损失曲面形状。
• 将暴力方法与标准剪枝技术（如Optimal Brain Damage和Optimal Brain Surgeon）进行对比，后者依赖于线性或二次误差近似。
• 通过测量移除一个无关神经元后其他神经元贡献的变化，分析神经元之间的相互依赖性，揭示网络的动态重构机制。
• 通过在已训练网络上的实证实验验证该方法，结果表明在仅造成极小精度退化的情况下，可移除40–70%的神经元。

实验结果

研究问题

• RQ1 神经网络中的学习表征在神经元之间分布是否不均？这如何挑战传统剪枝假设？
• RQ2 为何依赖线性或二次误差函数近似神经元重要性的标准剪枝算法无法做出最优剪枝决策？
• RQ3 在完全训练的网络中，'无关'神经元扮演什么角色？它们如何影响其他神经元的贡献？
• RQ4 暴力、串行的神经元剪枝方法能否实现高倍率压缩且性能损失极小？若是，为何其优于基于启发式的方法？
• RQ5 神经元之间的相互依赖性如何使'可独立剪枝'的假设失效？

主要发现

• 与剪枝单个权重相比，剪枝整个神经元可实现显著更高的压缩率（原始大小的40–70%），且在不微调的情况下精度下降极小。
• 暴力神经元剪枝方法优于依赖错误曲面线性或二次近似的标准技术（如Optimal Brain Damage），因为后者存在错误的近似假设。
• 该研究通过实证验证了Mozer & Smolensky于1989年的观察：仅少数'精英'神经元承载主要的学习表征，其余神经元起辅助作用。
• 无关神经元并非真正冗余；其移除会改变整个网络的动力学，表明神经元之间存在强烈的相互依赖关系。
• 误差曲面可视化显示，标准剪枝方法由于对真实损失曲面的过度简化近似，容易做出灾难性的剪枝决策。
• 尽管计算成本较高，暴力神经元剪枝具有高度可并行化特性，通过基于子集的近似方法可使其具备实际可行性。