[论文解读] A Surrogate Lagrangian Relaxation-based Model Compression for Deep Neural Networks.

Deniz Gurevin, Shanglin Zhou+5 more|arXiv (Cornell University)|Dec 18, 2020

Advanced Neural Network Applications被引用 4

一句话总结

本文提出了一种代理拉格朗日松弛（Surrogate Lagrangian Relaxation, SLR）方法，用于高效地剪枝深度神经网络权重，通过利用拉格朗日乘子和二次惩罚项加速收敛，从而避免了大量微调训练。SLR在极少微调的情况下实现了高精度模型——例如，在PointPillars模型上实现了9.44倍的压缩率，仅需3个训练周期，精度损失低于1%，在9.49倍压缩率下相比ADMM方法精度高出15.2%。

ABSTRACT

Network pruning is a widely used technique to reduce computation cost and model size for deep neural networks. However, the typical three-stage pipeline, i.e., training, pruning and retraining (fine-tuning) significantly increases the overall training trails. For instance, the retraining process could take up to 80 epochs for ResNet-18 on ImageNet, that is 70% of the original model training trails. In this paper, we develop a systematic weight-pruning optimization approach based on Surrogate Lagrangian relaxation (SLR), which is tailored to overcome difficulties caused by the discrete nature of the weight-pruning problem while ensuring fast convergence. We decompose the weight-pruning problem into subproblems, which are coordinated by updating Lagrangian multipliers. Convergence is then accelerated by using quadratic penalty terms. We evaluate the proposed method on image classification tasks, i.e., ResNet-18, ResNet-50 and VGG-16 using ImageNet and CIFAR-10, as well as object detection tasks, i.e., YOLOv3 and YOLOv3-tiny using COCO 2014, PointPillars using KITTI 2017, and Ultra-Fast-Lane-Detection using TuSimple lane detection dataset. Numerical testing results demonstrate that with the adoption of the Surrogate Lagrangian Relaxation method, our SLR-based weight-pruning optimization approach achieves a high model accuracy even at the hard-pruning stage without retraining for many epochs, such as on PointPillars object detection model on KITTI dataset where we achieve 9.44x compression rate by only retraining for 3 epochs with less than 1% accuracy loss. As the compression rate increases, SLR starts to perform better than ADMM and the accuracy gap between them increases. SLR achieves 15.2% better accuracy than ADMM on PointPillars after pruning under 9.49x compression. Given a limited budget of retraining epochs, our approach quickly recovers the model accuracy.

研究动机与目标

• 为解决深度神经网络压缩中标准三阶段训练-剪枝-微调流程带来的高计算成本问题。
• 克服权重剪枝的离散性，该特性阻碍了在连续空间中的高效优化。
• 开发一种能够加速收敛并在极端剪枝条件下仍保持高模型精度的方法。
• 减少剪枝后所需的微调训练周期数，特别是在训练预算有限的情况下。

提出的方法

• 将权重剪枝问题建模为使用拉格朗日松弛处理稀疏性约束的约束优化问题。
• 引入代理松弛技术，以处理二值剪枝决策的不可微性。
• 将主问题分解为子问题，通过拉格朗日乘子的对偶上升法进行迭代求解与协调。
• 采用二次惩罚项以提升优化过程中的收敛速度与稳定性。
• 通过在网络权重与拉格朗日乘子之间交替优化，联合最小化松弛后的目标函数。
• 在多种架构（ResNet、VGG、YOLO、PointPillars）和数据集（ImageNet、COCO、KITTI、TuSimple）上应用该方法，以验证其泛化能力。

实验结果

研究问题

• RQ1 代理拉格朗日松弛方法是否能在极少微调训练周期下实现高精度的模型压缩？
• RQ2 在高剪枝率下，SLR与ADMM在精度和收敛速度方面相比如何？
• RQ3 在未进行大量微调的情况下，SLR在极端剪枝条件下能多大程度上保持模型性能？
• RQ4 SLR是否能在包括图像分类与目标检测在内的多种架构与任务中实现良好泛化？
• RQ5 SLR是否能在保持竞争力精度的同时减少所需微调训练周期数？

主要发现

• 在KITTI 2017数据集上的PointPillars模型中，SLR实现了9.44倍的压缩率，仅需3个微调训练周期，精度损失低于1%。
• 在9.49倍压缩率下，SLR在PointPillars模型上的精度比ADMM高出15.2%。
• SLR在硬剪枝阶段仍能保持高精度，显著减少了对长时间微调的需求。
• 随着压缩率的提高，SLR在性能上持续优于ADMM，且在更高稀疏度下精度差距进一步扩大。
• 在ImageNet上的ResNet-18模型中，SLR将微调时间减少至原始训练时间的30%以下，避免了标准流程中常见的80个训练周期的微调。
• SLR在包括图像分类（ImageNet、CIFAR-10）和目标检测（COCO、KITTI、TuSimple）在内的多种任务中均表现出强大的泛化能力。