[Paper Review] A novel channel pruning method for deep neural network compression

Yiming Hu, Siyang Sun+3 more|arXiv (Cornell University)|May 29, 2018

Anomaly Detection Techniques and Applications30 references65 citations

TL;DR

This paper presents a genetic-algorithm-based channel pruning method for pre-trained CNNs, using layer-wise pruning with Hessian-based fitness and knowledge-distillation fine-tuning to achieve compact models with minimal accuracy loss.

ABSTRACT

In recent years, deep neural networks have achieved great success in the field of computer vision. However, it is still a big challenge to deploy these deep models on resource-constrained embedded devices such as mobile robots, smart phones and so on. Therefore, network compression for such platforms is a reasonable solution to reduce memory consumption and computation complexity. In this paper, a novel channel pruning method based on genetic algorithm is proposed to compress very deep Convolution Neural Networks (CNNs). Firstly, a pre-trained CNN model is pruned layer by layer according to the sensitivity of each layer. After that, the pruned model is fine-tuned based on knowledge distillation framework. These two improvements significantly decrease the model redundancy with less accuracy drop. Channel selection is a combinatorial optimization problem that has exponential solution space. In order to accelerate the selection process, the proposed method formulates it as a search problem, which can be solved efficiently by genetic algorithm. Meanwhile, a two-step approximation fitness function is designed to further improve the efficiency of genetic process. The proposed method has been verified on three benchmark datasets with two popular CNN models: VGGNet and ResNet. On the CIFAR-100 and ImageNet datasets, our approach outperforms several state-of-the-art methods. On the CIFAR-10 and SVHN datasets, the pruned VGGNet achieves better performance than the original model with 8 times parameters compression and 3 times FLOPs reduction.

Motivation & Objective

• Motivate model compression for deploying CNNs on resource-constrained devices.
• Develop a channel pruning method that treats channel selection as a combinatorial optimization problem.
• Reduce model redundancy while preserving accuracy through informed pruning and effective fine-tuning.
• Explore layer-wise pruning with sensitivity-based per-layer compression rates and knowledge distillation for recovery.

Proposed method

• Encode convolution kernels as binary chromosomes representing channel retention decisions.
• Apply a genetic algorithm with elitism, roulette-wheel selection, single-point crossover, and mutation to search the channel subset.
• Use a three-term, fast-efficiency fitness function based on a layer-wise error approximation and a second-order Taylor expansion to estimate pruning impact.
• Approximate the Hessian via H = (1/N) sum_i X_i X_i^T to compute channel sensitivities for pruning.
• Prune layers sequentially, using layer sensitivity to group layers and assign groupwise compression rates, followed by fine-tuning with knowledge distillation.
• Fine-tune pruned networks with a teacher (original) model using attention-map alignment to guide student learning.

Experimental results

Research questions

• RQ1 Can a genetic algorithm effectively search the exponential channel-pruning space to identify compact CNN architectures without large accuracy loss?
• RQ2 Does a Hessian-informed, second-order fitness function improve pruning efficiency and final accuracy compared to first-order or heuristic criteria?
• RQ3 How does per-layer (grouped) sensitivity-based pruning affect the performance/compression trade-off across VGGNet and ResNet on CIFAR, SVHN, and ImageNet?
• RQ4 Does knowledge-distillation-based fine-tuning improve recovery after pruning, compared to standard fine-tuning?
• RQ5 Is pruning compatible with layer-by-layer sequential removal and subsequent fine-tuning for both CNN backbones (VGGNet, ResNet) on multiple datasets?

Key findings

• The proposed genetic-algorithm channel pruning method outperforms several state-of-the-art channel-selection strategies on CIFAR-100 and ImageNet.
• On CIFAR-10 and SVHN, the pruned VGGNet achieves substantial parameter and FLOP reductions with maintained or improved accuracy.
• Layer-wise pruning with sensitivity grouping enables aggressive pruning in deeper, more redundant layers while safeguarding accuracy in sensitive layers.
• Knowledge-distillation-based fine-tuning (FT-KD) provides measurable accuracy gains over standard fine-tuning, especially at higher compression rates.
• The approach demonstrates effective compression on both VGGNet and ResNet across multiple data domains, including CIFAR, SVHN, and ImageNet.