[論文レビュー] A Survey of Model Compression and Acceleration for Deep Neural Networks

Yu Cheng, Duo Wang+2 more|arXiv (Cornell University)|Oct 23, 2017

Anomaly Detection Techniques and Applications参考文献 81被引用数 878

ひとこと要約

この調査は、深層ニューラルネットワークを圧縮・加速する最近の技術を、剪定/量子化、低秩分解、転送/コンパクト畳み込みフィルター、知識蒸留の4カテゴリに分類し、ベンチマークと将来の課題について議論します。

ABSTRACT

Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.

研究の動機と目的

• Identify and categorize major model compression and acceleration techniques for deep neural networks.
• Analyze the strengths, drawbacks, and typical applications of each category.
• Survey training protocols (pre-trained vs from-scratch) and end-to-end versus modular approaches.
• Summarize evaluation metrics, datasets, and benchmarks used in compression literature.
• Discuss challenges and potential directions for future research.

提案手法

• Categorize approaches into four main groups: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation.
• Describe sub-techniques within each category (e.g., quantization/binarization, structured sparsity, Hessian-based pruning, CP/BN-based low-rank decompositions, adaptive/structured matrices, and teacher-student distillation).
• Explain training paradigms (pre-trained pruning/quantization vs from-scratch training for transfer/compact filters and distillation).
• Present evaluation criteria (compression rate, speedup, and accuracy) and discuss practical deployment aspects across CPU/GPU and hardware.
• Summarize representative benchmarks and baseline models used in compression research (e.g., AlexNet, VGG, GoogleNet, ResNet) and performance tables where provided.]
• research_questions:[

実験結果

リサーチクエスチョン

• RQ1 What are the main categories of model compression and acceleration for DNNs and how do they differ in applicability and impact?
• RQ2 How do pruning/quantization, low-rank factorization, transferred filters, and knowledge distillation compare in terms of accuracy, compression, and speedup across common architectures?
• RQ3 What evaluation metrics, datasets, and benchmarks best capture compression performance, and what are the typical trade-offs?
• RQ4 What are the remaining challenges and promising directions for future work in DNN model compression?
• RQ5 How should one choose an appropriate compression approach for a given application and hardware constraint?

主な発見

Model	TOP-5 Accuracy	Speed-up	Compression Rate
AlexNet	80.03%	1.	1.
BN Low-rank	80.56%	1.09	4.94
CP Low-rank	79.66%	1.82	5.
VGG-16	90.60%	1.	1.
BN Low-rank	90.47%	1.53	2.72
CP Low-rank	90.31%	2.05	2.75
GoogleNet	92.21%	1.	1.
BN Low-rank	91.88%	1.08	2.79
CP Low-rank	91.79%	1.20	2.84

• Four primary categories capture the landscape: pruning/quantization, low-rank factorization, transferred/compact filters, and knowledge distillation.
• These methods are largely orthogonal and can be combined (e.g., pruning with quantization or low-rank with transferred filters).
• Transferred/compact filters can reduce parameters for convolutional layers but depend on architectural choices and may not suit very deep/thin networks as well as other methods.
• Knowledge distillation can yield compact networks that mimic larger teachers, but may be less competitive and often task-dependent.
• Low-rank methods offer straightforward compression but typically involve layer-wise decompositions and retraining, with potential difficulties in global optimization.
• Benchmarking commonly uses networks like AlexNet, VGG, GoogleNet, and ResNet, with metrics on compression rate, speedup, and accuracy.