[Paper Review] A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions

Rahul Mishra, Hari Prabhat Gupta+1 more|arXiv (Cornell University)|Oct 5, 2020

Advanced Neural Network ApplicationsComputer Science97 references60 citations

TL;DR

This paper surveys existing deep neural network (DNN) compression techniques, categorizing them into pruning, sparse representations, low-precision, knowledge distillation, and miscellaneous, and discusses challenges and future directions for IoT deployment.

ABSTRACT

Deep Neural Network (DNN) has gained unprecedented performance due to its automated feature extraction capability. This high order performance leads to significant incorporation of DNN models in different Internet of Things (IoT) applications in the past decade. However, the colossal requirement of computation, energy, and storage of DNN models make their deployment prohibitive on resource constraint IoT devices. Therefore, several compression techniques were proposed in recent years for reducing the storage and computation requirements of the DNN model. These techniques on DNN compression have utilized a different perspective for compressing DNN with minimal accuracy compromise. It encourages us to make a comprehensive overview of the DNN compression techniques. In this paper, we present a comprehensive review of existing literature on compressing DNN model that reduces both storage and computation requirements. We divide the existing approaches into five broad categories, i.e., network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous, based upon the mechanism incorporated for compressing the DNN model. The paper also discussed the challenges associated with each category of DNN compression techniques. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.

Motivation & Objective

• Provide a thorough overview of DNN compression techniques to enable deployment on resource-constrained IoT devices.
• Categorize compression methods into five broad groups and map representative works within each category.
• Identify challenges and gaps in current techniques to suggest future research directions.
• Summarize how each category mitigates storage, computation, and energy requirements while preserving accuracy.

Proposed method

• Classify DNN compression techniques into five categories: network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous.
• Review sub-categories within each category (e.g., channel/filter/connection/layer pruning; quantization, multiplexing, weight sharing; integer estimation, low-bit representations, binarization; logits transfer, teacher assistant, domain adaptation).
• Discuss challenges associated with each category, including accuracy trade-offs and deployment considerations on resource-constrained devices.
• Provide a consolidated summary of existing work under each category and outline future directions for DNN compression.

Experimental results

Research questions

• RQ1 What are the main categories and sub-categories of DNN compression techniques in the literature?
• RQ2 What are the key challenges and accuracy trade-offs associated with each compression category?
• RQ3 How do current approaches address deployment on resource-constrained IoT devices, and where are the gaps for future work?
• RQ4 What future directions can advance DNN compression to achieve better storage, computation, and energy efficiency without significant accuracy loss?

Key findings

• DNN compression literature is organized into five broad categories: network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous.
• Channel, filter, connection, and layer pruning are the main sub-categories of network pruning, each with distinct strategies and trade-offs.
• Sparse representation includes quantization, multiplexing, and weight sharing aimed at reducing storage and FLOPs while preserving performance.
• Bits precision techniques cover integer estimation, low-bit representations, and binarization to lower weight storage and computation.
• Knowledge distillation transfers generalization from a large teacher model to a smaller student model to mitigate accuracy loss after compression.
• Miscellaneous techniques focus on deployment aspects such as mobile and embedded device suitability and parallelization.