[Paper Review] How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference

Tai Vu, Emily Wen+1 more|arXiv (Cornell University)|Mar 30, 2020

Image Enhancement Techniques31 references4 citations

TL;DR

This paper proposes combining mixup and cutout regularization with soft filter pruning in ResNet architectures to simultaneously reduce FLOPs and improve inference efficiency without sacrificing accuracy. The combination achieves up to 14.8% FLOP reduction and up to 2.03% accuracy gain over baseline models, outperforming either technique alone.

ABSTRACT

The challenge of speeding up deep learning models during the deployment phase has been a large, expensive bottleneck in the modern tech industry. In this paper, we examine the use of both regularization and pruning for reduced computational complexity and more efficient inference in Deep Neural Networks (DNNs). In particular, we apply mixup and cutout regularizations and soft filter pruning to the ResNet architecture, focusing on minimizing floating-point operations (FLOPs). Furthermore, by using regularization in conjunction with network pruning, we show that such a combination makes a substantial improvement over each of the two techniques individually.

Motivation & Objective

• To address the computational inefficiency of deep neural networks during inference, especially on resource-constrained devices.
• To investigate whether regularization and pruning can be combined synergistically rather than competitively.
• To evaluate the impact of regularization on computational cost, a largely unexplored area in prior work.
• To achieve significant FLOP reduction with minimal accuracy drop, enabling efficient deployment of DNNs.
• To validate that the combination of regularization and pruning yields better performance than either method applied individually.

Proposed method

• Applied mixup and cutout regularization during training to improve generalization and robustness.
• Used soft filter pruning to remove low-magnitude convolutional filters, reducing parameter count and FLOPs.
• Trained ResNet architectures (20, 32, 44, 56, 110) on CIFAR-10 with combined regularization and pruning.
• Set pruning rate at 0.9 (10% of filters retained per layer) to achieve substantial FLOP reduction.
• Compared performance across control, regularization-only, pruning-only, and combined models using FLOPs and accuracy as metrics.
• Evaluated models on standard benchmarks to measure trade-offs between computational cost and accuracy.

Experimental results

Research questions

• RQ1 Can regularization techniques like mixup and cutout reduce computational complexity during inference, or do they introduce overhead?
• RQ2 Does combining regularization with pruning lead to better performance than using either technique in isolation?
• RQ3 To what extent can soft filter pruning reduce FLOPs in ResNet architectures without degrading accuracy?
• RQ4 How do different pruning rates affect the synergy between regularization and pruning?
• RQ5 Can the combined approach maintain or improve accuracy while significantly reducing FLOP count?

Key findings

• The combination of mixup and pruning increased accuracy by up to 1.31% on ResNet-56 while reducing FLOPs by 14.7%.
• Cutout and pruning together improved accuracy by up to 2.03% on ResNet-56, with a 14.7% reduction in FLOPs.
• Soft filter pruning reduced FLOPs by approximately 15% across all ResNet sizes, even with a pruning rate of 0.9.
• Models using both regularization and pruning achieved higher accuracy than pruning-only models, demonstrating synergy.
• The combined method reduced FLOPs by ~15% compared to regularization-only models, with accuracy either maintained or slightly improved.
• The results validate that regularization and pruning can be effectively combined to enhance both efficiency and accuracy in DNN inference.