[논문 리뷰] A Survey of Model Compression and Acceleration for Deep Neural Networks

Yu Cheng, Duo Wang외 2명|arXiv (Cornell University)|2017. 10. 23.

Anomaly Detection Techniques and Applications참고 문헌 81인용 수 878

한 줄 요약

이 설문은 최근의 딥 뉴럴 네트워크 압축 및 가속 기술을 정리하고, 가지를 prune/quantization, low-rank factorization, transferred/compact filters, knowledge distillation로 분류하며 벤치마크와 향후 과제를 논의한다.

ABSTRACT

Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.

연구 동기 및 목표

• Identify and categorize major model compression and acceleration techniques for deep neural networks.
• Analyze the strengths, drawbacks, and typical applications of each category.
• Survey training protocols (pre-trained vs from-scratch) and end-to-end versus modular approaches.
• Summarize evaluation metrics, datasets, and benchmarks used in compression literature.
• Discuss challenges and potential directions for future research.

제안 방법

• Categorize approaches into four main groups: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation.
• Describe sub-techniques within each category (e.g., quantization/binarization, structured sparsity, Hessian-based pruning, CP/BN-based low-rank decompositions, adaptive/structured matrices, and teacher-student distillation).
• Explain training paradigms (pre-trained pruning/quantization vs from-scratch training for transfer/compact filters and distillation).
• Present evaluation criteria (compression rate, speedup, and accuracy) and discuss practical deployment aspects across CPU/GPU and hardware.
• Summarize representative benchmarks and baseline models used in compression research (e.g., AlexNet, VGG, GoogleNet, ResNet) and performance tables where provided.]
• research_questions: [

실험 결과

연구 질문

• RQ1 DNN의 모델 압축 및 가속에 대한 주요 카테고리는 무엇이며, 적용성 및 영향 측면에서 어떻게 다른가?
• RQ2 pruning/quantization, low-rank factorization, transferred filters, knowledge distillation은 일반 아키텍처에서 정확도, 압축률, 속도향상 측면에서 어떻게 비교되는가?
• RQ3 압축 성능을 가장 잘 포착하는 평가 지표, 데이터셋 및 벤치마크는 무엇이며, 일반적인 트레이드오프는 무엇인가?
• RQ4 DNN 모델 압축의 남은 도전과제 및 미래 연구를 위한 유망한 방향은 무엇인가?
• RQ5 주어진 애플리케이션과 하드웨어 제약에 대해 적합한 압축 방식을 어떻게 선택해야 하는가?

주요 결과

• 네 가지 주요 카테고리가 현재 landscape를 포착한다: pruning/quantization, low-rank factorization, transferred/compact filters, knowledge distillation.
• 이 방법들은 거의 직교적이며 결합하여 사용할 수 있다(예: pruning과 quantization, 또는 low-rank와 transferred filters).
• Transferred/compact filters는 합성곱 계층의 파라미터를 줄일 수 있지만 아키텍처 선택에 의존하고 매우 깊거나 얇은 네트워크에는 다른 방법만큼 잘 맞지 않을 수 있다.
• Knowledge distillation은 더 큰 teacher를 모방하는 컴팩트한 네트워크를 만들 수 있지만, 경쟁력이 떨어지거나 과제에 따라 달라질 수 있다.
• 저랭크 방법은 직관적인 압축을 제공하지만 일반적으로 계층별 분해 및 재학습이 필요하고, 전역 최적화에서의 어려움이 있을 수 있다.
• 벤치마크링은 일반적으로 AlexNet, VGG, GoogleNet, ResNet 같은 네트워크를 사용하며 압축률, 속도향상, 정확도에 대한 지표를 제시한다.