[论文解读] Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks

Xiangyu Chang, Yingcong Li+2 more|arXiv (Cornell University)|Dec 16, 2020

Sparse and Compressive Sensing Techniques被引用 4

一句话总结

本文对模型压缩中的过参数化现象提供了理论分析，表明在剪枝前训练大模型可获得比从零开始训练小模型更好的泛化性能，即使最优特征已知。该研究在剪枝模型中建立了新颖的双下降曲线，并证明仅当特征相关时，微调才有益处。

ABSTRACT

Deep networks are typically trained with many more parameters than the size of the training dataset. Recent empirical evidence indicates that the practice of overparameterization not only benefits training large models, but also assists - perhaps counterintuitively - building lightweight models. Specifically, it suggests that overparameterization benefits model pruning / sparsification. This paper sheds light on these empirical findings by theoretically characterizing the high-dimensional asymptotics of model pruning in the overparameterized regime. The theory presented addresses the following core question: "should one train a small model from the beginning, or first train a large model and then prune?". We analytically identify regimes in which, even if the location of the most informative features is known, we are better off fitting a large model and then pruning rather than simply training with the known informative features. This leads to a new double descent in the training of sparse models: growing the original model, while preserving the target sparsity, improves the test accuracy as one moves beyond the overparameterization threshold. Our analysis further reveals the benefit of retraining by relating it to feature correlations. We find that the above phenomena are already present in linear and random-features models. Our technical approach advances the toolset of high-dimensional analysis and precisely characterizes the asymptotic distribution of over-parameterized least-squares. The intuition gained by analytically studying simpler models is numerically verified on neural networks.

研究动机与目标

• 理解为何在剪枝前训练大模型能改善泛化性能，尽管模型规模增加。
• 通过理论分析解决过参数化是否有助于构建轻量化、稀疏模型的开放问题。
• 在高维渐近条件下表征剪枝模型的泛化性能，特别是过参数化区域的表现。
• 研究微调在剪枝中的作用，特别是特征相关性如何影响其有效性。
• 提出过参数化最小二乘法的分布表征（DC），以准确预测剪枝结果。

提出的方法

• 引入过参数化最小二乘回归解的分布表征（DC），实现对剪枝后测试误差的精确预测。
• 利用线性高斯等价性，将线性模型的理论结果扩展至非线性随机特征模型。
• 通过推导剪枝并微调后模型权重的渐近分布，分析训练-剪枝-微调流程。
• 将剪枝模型的风险表述为稀疏度与过参数化程度的函数，识别出测试误差中的双下降行为。
• 利用推导出的分布模型量化特征相关性对微调性能的影响。
• 通过在随机特征模型和CIFAR-10上的ResNet-20上进行大量数值模拟，验证理论预测。

实验结果

研究问题

• RQ1 即使已知最优稀疏结构，过参数化是否仍能提升剪枝模型的泛化性能？
• RQ2 当模型宽度超过过参数化阈值后，剪枝模型的测试误差中是否会出现双下降曲线？
• RQ3 在何种条件下剪枝后的微调是有益的？特征相关性如何影响这一效果？
• RQ4 过参数化最小二乘解的分布与剪枝性能有何关系？
• RQ5 线性模型的理论预测能否准确描述非线性神经网络中的剪枝动态？

主要发现

• 即使已知最优支持集，剪枝过参数化模型的测试误差仍低于使用相同最优特征训练的小模型。
• 在剪枝模型的测试误差中出现了一种新颖的双下降行为：当模型宽度超过过参数化阈值后，风险随宽度增加而降低。
• 当特征不相关时，剪枝后微调的额外风险为固定值 $ \sigma^2 / (n-2) $，与剪枝集合无关。
• 剪枝引起的误差中的偏差项被量化为 $ (1 - \zeta)^2 c^{\star 2} $，其中 $ \zeta $ 为剪枝后保留的信号比例。
• 理论DC在随机特征模型和CIFAR-10上的ResNet-20中均能准确预测剪枝性能，验证了其向深度网络的可迁移性。
• 特征相关性决定了微调的有效性：当相关性高时，微调至关重要；但当特征不相关时，微调可能损害性能。