[论文解读] ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations

Rishabh Tiwari, Udbhav Bamba+2 more|arXiv (Cornell University)|Feb 14, 2021

Advanced Neural Network Applications参考文献 35被引用 31

一句话总结

ChipNet 提出了一种确定性、预算感知的结构化裁剪方法，使用连续的 Heaviside 投影和脆性损失，从预训练的密集模型中获得高度稀疏的裁剪网络，在多种预算和数据集上优于最先进的基线。

ABSTRACT

Structured pruning methods are among the effective strategies for extracting small resource-efficient convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network. Our choice of continuous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible and expected as final outcomes. Our approach's flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.

研究动机与目标

• 推动稳健的结构化裁剪，能够在不从头重新训练的情况下强制执行任意预算约束。
• 开发一种裁剪机制，使用梯度优化从密集网络中产生接近离散的通道掩码。
• 在极低预算条件下实现稳定性，并证明学到的掩码在跨数据集上的可迁移性。
• 显示在多种预算下，ChipNet 相较于现有裁剪方法具有更高的准确性。

提出的方法

• 为预训练的密集 CNN 的通道学习稀疏性掩码。
• 使用连续的 Heaviside 投影结合逻辑代理，将掩码驱动到 0 或 1。
• 引入一个脆性损失，惩罚中间掩码值以鼓励离散掩码。
• 通过预算损失施加预算约束，该损失可适应通道、激活-体积、参数或 FLOPs 的预算。
• 使用包含交叉熵、脆性和预算项的联合损失进行训练；先进行软裁剪，再通过二值掩码进行硬裁剪。
• 允许预算函数 V 取若干选项中的任意一个（通道、体积、参数、FLOPs），并在预算阶段对掩码的逻辑投影版本进行估计。

实验结果

研究问题

• RQ1 一个确定性、预算感知的裁剪方法是否能够在不从头重新训练的情况下产生高度稀疏的网络？
• RQ2 连续的 Heaviside 投影再加上脆性损失是否能产生接近离散的掩码，以满足多样的预算约束？
• RQ3 学到的裁剪掩码是否可跨数据集和任务领域迁移？
• RQ4 在多种预算形式下，ChipNet 相对于最先进的预算感知裁剪方法的表现如何？
• RQ5 ChipNet 在非常低资源预算（如极端裁剪）下是否稳定有效？

主要发现

• ChipNet 在不同预算类型和数据集上持续超越最先进的裁剪基线。
• 该方法在极低预算甚至极端裁剪场景下仍然稳定且有效。
• ChipNet 学到的掩码可以跨数据集迁移，有时甚至优于直接在目标数据上学习的掩码。
• 在若干实验中，在相似预算下，ChipNet 相对于基线实现了显著的精度提升。
• 从特征丰富的教师数据集迁移掩码，往往比仅在学生数据上裁剪得到的学生模型表现更佳。