[Paper Review] Deep Model Compression based on the Training History

S. H. Shabbeer Basha, Mohammad Farazuddin+3 more|arXiv (Cornell University)|Jan 30, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes History Based Filter Pruning (HBFP), a novel filter pruning method that leverages training history to identify and remove redundant filters by analyzing the similarity of their ℓ₁-norms across epochs. By optimizing filter pairs to minimize information loss before pruning and fine-tuning afterward, HBFP achieves up to 97.98% FLOPs reduction on LeNet-5 while maintaining low error rates, outperforming state-of-the-art methods.

ABSTRACT

Deep Convolutional Neural Networks (DCNNs) have shown promising performances in several visual recognition problems which motivated the researchers to propose popular architectures such as LeNet, AlexNet, VGGNet, ResNet, and many more. These architectures come at a cost of high computational complexity and parameter storage. To get rid of storage and computational complexity, deep model compression methods have been evolved. We propose a "History Based Filter Pruning (HBFP)" method that utilizes network training history for filter pruning. Specifically, we prune the redundant filters by observing similar patterns in the filter's L1-norms (absolute sum of weights) over the training epochs. We iteratively prune the redundant filters of a CNN in three steps. First, we train the model and select the filter pairs with redundant filters in each pair. Next, we optimize the network to ensure an increased measure of similarity between the filters in a pair. This optimization of the network facilitates us to prune one filter from each pair based on its importance without much information loss. Finally, we retrain the network to regain the performance, which is dropped due to filter pruning. We test our approach on popular architectures such as LeNet-5 on MNIST dataset; VGG-16, ResNet-56, and ResNet-110 on CIFAR-10 dataset, and ResNet-50 on ImageNet. The proposed pruning method outperforms the state-of-the-art in terms of FLOPs reduction (floating-point operations) by 97.98%, 83.42%, 78.43%, 74.95%, and 75.45% for LeNet-5, VGG-16, ResNet-56, ResNet-110, and ResNet-50, respectively, while maintaining the less error rate.

Motivation & Objective

• To address the high computational and storage costs of deep CNNs on resource-constrained devices.
• To improve filter pruning efficiency by leveraging training dynamics rather than static metrics.
• To minimize information loss during pruning by optimizing filter similarity before removal.
• To achieve higher compression rates than existing methods while preserving model accuracy.
• To validate the method across diverse architectures and datasets, including ImageNet.

Proposed method

• The method identifies redundant filter pairs by measuring the sum of absolute differences in their ℓ₁-norms across training epochs.
• A custom regularizer is introduced to minimize the difference between ℓ₁-norms of filter pairs, increasing their redundancy and reducing information loss upon pruning.
• One filter from each redundant pair is pruned based on its ℓ₁-norm magnitude, favoring the less important filter.
• The network is fine-tuned after each pruning iteration to recover performance degraded by filter removal.
• The process is repeated iteratively to achieve progressive compression.
• The approach is adaptable to other metrics such as ℓ₂-norm or cosine similarity.

Experimental results

Research questions

• RQ1 Can training history be effectively used to identify redundant filters for pruning without significant performance drop?
• RQ2 Does optimizing filter similarity before pruning reduce information loss compared to direct pruning?
• RQ3 Can the proposed method achieve higher FLOPs reduction than state-of-the-art filter pruning techniques?
• RQ4 How does the method perform across different architectures and datasets, including large-scale ImageNet?
• RQ5 Does the use of a custom regularizer significantly improve post-pruning accuracy?

Key findings

• HBFP achieved a 97.98% reduction in FLOPs on LeNet-5 with minimal accuracy drop, outperforming prior methods.
• On VGG-16, the method reduced FLOPs by 83.42% while maintaining high accuracy, surpassing state-of-the-art results.
• For ResNet-56 and ResNet-110 on CIFAR-10, HBFP achieved 78.43% and 74.95% FLOPs reduction, respectively, with competitive accuracy.
• On ImageNet using ResNet-50, HBFP reduced FLOPs by 75.45% and parameters by 68.30%, achieving 69.17% top-1 accuracy.
• The ablation study confirmed that the custom regularizer significantly improves performance by minimizing information loss during pruning.
• HBFP demonstrated robustness across low- and high-resolution datasets, including MNIST, CIFAR-10, and ImageNet.