[论文解读] Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks

Melih Barsbey, Milad Sefidgaran+3 more|arXiv (Cornell University)|Jun 7, 2021

Stochastic Gradient Optimization Techniques参考文献 55被引用 12

一句话总结

本文建立了随机梯度下降（SGD）中权重分布的重尾特性与过参数化神经网络可压缩性之间的理论联系。证明了当SGD表现出重尾平稳分布（由于学习率与小批量大小之比较大）且混沌传播出现（由于过参数化）时，网络可被严格证明为ℓₚ-可压缩，且随着模型规模增大，剪枝误差趋于消失——为简单剪枝为何有效以及可压缩网络为何泛化性能良好提供了统一解释。

ABSTRACT

Neural network compression techniques have become increasingly popular as they can drastically reduce the storage and computation requirements for very large networks. Recent empirical studies have illustrated that even simple pruning strategies can be surprisingly effective, and several theoretical studies have shown that compressible networks (in specific senses) should achieve a low generalization error. Yet, a theoretical characterization of the underlying cause that makes the networks amenable to such simple compression schemes is still missing. In this study, we address this fundamental question and reveal that the dynamics of the training algorithm has a key role in obtaining such compressible networks. Focusing our attention on stochastic gradient descent (SGD), our main contribution is to link compressibility to two recently established properties of SGD: (i) as the network size goes to infinity, the system can converge to a mean-field limit, where the network weights behave independently, (ii) for a large step-size/batch-size ratio, the SGD iterates can converge to a heavy-tailed stationary distribution. In the case where these two phenomena occur simultaneously, we prove that the networks are guaranteed to be '$\ell_p$-compressible', and the compression errors of different pruning techniques (magnitude, singular value, or node pruning) become arbitrarily small as the network size increases. We further prove generalization bounds adapted to our theoretical framework, which indeed confirm that the generalization error will be lower for more compressible networks. Our theory and numerical study on various neural networks show that large step-size/batch-size ratios introduce heavy-tails, which, in combination with overparametrization, result in compressibility.

研究动机与目标

• 识别使过参数化神经网络适用于简单剪枝策略的根本机制。
• 解释为何可压缩网络泛化性能良好，填补了当前对过参数化模型泛化性理论理解的空白。
• 将深度学习中两个近期现象——重尾SGD动力学与混沌传播——统一到一个解释可压缩性的框架中。
• 在特定训练超参数配置（大学习率/小批量大小比值）下，为ℓₚ-可压缩性提供理论保证。
• 推导出与ℓₚ-可压缩性框架适配的泛化误差界，将可压缩性与低泛化误差联系起来。

提出的方法

• 分析过参数化网络中SGD的平均场极限，表明随着网络规模增大，权重向量趋于独立。
• 证明当学习率/小批量大小比值较大时，SGD迭代收敛至重尾平稳分布。
• 结合混沌传播（权重独立）与重尾分布，证明全连接网络的ℓₚ-可压缩性。
• 利用压缩感知理论的结果，对基于大小、奇异值和节点的剪枝方法的压缩误差进行界定。
• 推导出适配于ℓₚ-可压缩性框架的泛化误差界，将可压缩性与低泛化误差联系起来。
• 利用高维球面上的集中与熵界，控制量化权重配置的数量，从而实现误差分析。

实验结果

研究问题

• RQ1 为何简单剪枝策略在过参数化神经网络中如此有效？
• RQ2 SGD的训练动力学——特别是重尾权重分布——在神经网络可压缩性中扮演何种角色？
• RQ3 过参数化如何与SGD超参数相互作用以实现可压缩性？
• RQ4 在特定训练条件下，ℓₚ-可压缩性能否被形式化保证？
• RQ5 可压缩性是否意味着泛化性能提升，且能否从理论上进行量化？

主要发现

• 当学习率/小批量大小比值较大时，SGD产生重尾权重分布，这是可压缩性的关键促成因素。
• 在过参数化状态下，混沌传播确保网络权重行为独立，从而支持有效剪枝。
• 当重尾分布与混沌传播同时出现时，随着网络规模增大，ℓₚ-可压缩性可被严格证明。
• 无论压缩比如何，基于大小、奇异值和节点的剪枝误差均随模型规模增大而趋于任意小。
• 更可压缩的网络具有更低的泛化误差，这一结论由推导出的泛化误差界所证实，且与经验观察一致。
• 在全连接与卷积网络上的数值实验验证了理论，显示重尾动力学与可压缩性之间存在强烈一致性。