[论文解读] Structured Pruning for Deep Convolutional Neural Networks: A survey

Yang He, Lingao Xiao|arXiv (Cornell University)|Mar 1, 2023

Advanced Neural Network Applications参考文献 287被引用 7

一句话总结

本论文综述了深度卷积神经网络的结构化剪枝方法，按滤波器排序、正则化、动态执行、神经架构搜索及相关扩展进行组织，并讨论未来方向。

ABSTRACT

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at https://github.com/he-y/Awesome-Pruning . A dedicated website offering a more interactive comparison of structured pruning methods can be found at: https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey .

研究动机与目标

• Explain why structured pruning is essential for deploying CNNs on resource-limited hardware.
• Categorize and compare existing structured pruning techniques across key dimensions such as ranking criteria, regularization, dynamics, and neural architecture search.
• Highlight connections to unstructured pruning and discuss practical considerations and future opportunities in structured pruning.

提出的方法

• Define a formal pruning objective that minimizes loss subject to a target filter budget.
• Present a taxonomy of structured pruning methods, including weight-dependent, activation-based, regularization-based, and optimization-based approaches, plus dynamic and NAS-inspired techniques.
• Discuss BN-based gating and extra-parameter gate methods to induce structured sparsity.
• Summarize representative methods and their core ideas across multiple subsections (2.1–2.7).
• Offer insights into future directions and potential research opportunities in structured pruning (Section 3).

实验结果

研究问题

• RQ1 What are effective criteria for ranking and selecting filters under structured pruning?
• RQ2 How can regularization and optimization techniques be used to induce structured sparsity while preserving accuracy?
• RQ3 How do dynamic pruning and NAS-based strategies contribute to practical CNN compression?
• RQ4 What are the potential future directions for structured pruning in light of evolving architectures such as Transformers?
• RQ5 How do structured pruning methods compare to unstructured pruning in terms of hardware efficiency and practicality?

主要发现

• The survey aggregates over 200 structured pruning papers and provides a unified taxonomy of methods.
• Structured pruning can be achieved via weight-dependent, activation-based, regularization-based, optimization-based, dynamic, and NAS-inspired techniques.
• BN-based gating and extra-parameter gates are effective for inducing channel- or filter-level sparsity with controllable budgets.
• Dynamic pruning during training or inference offers additional flexibility for maintaining accuracy under compression.
• Extensions of pruning include Lottery Ticket-inspired ideas, joint compression, and special granularity strategies to accommodate hardware and task needs.