[Paper Review] Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers

Chengcheng Li, Zi Wang+2 more|arXiv (Cornell University)|Feb 18, 2019

Domain Adaptation and Few-Shot Learning23 references4 citations

TL;DR

This paper proposes a single-shot channel pruning method using Alternating Direction Method of Multipliers (ADMM) to achieve filter-level sparsity during training, enabling one-time pruning and fine-tuning. It outperforms state-of-the-art iterative pruning methods, including Taylor expansion, across multiple architectures and datasets, especially at high compression ratios.

ABSTRACT

Channel pruning has been identified as an effective approach to constructing efficient network structures. Its typical pipeline requires iterative pruning and fine-tuning. In this work, we propose a novel single-shot channel pruning approach based on alternating direction methods of multipliers (ADMM), which can eliminate the need for complex iterative pruning and fine-tuning procedure and achieve a target compression ratio with only one run of pruning and fine-tuning. To the best of our knowledge, this is the first study of single-shot channel pruning. The proposed method introduces filter-level sparsity during training and can achieve competitive performance with a simple heuristic pruning criterion (L1-norm). Extensive evaluations have been conducted with various widely-used benchmark architectures and image datasets for object classification purpose. The experimental results on classification accuracy show that the proposed method can outperform state-of-the-art network pruning works under various scenarios.

Motivation & Objective

• To eliminate the need for iterative pruning and fine-tuning in channel pruning by introducing a single-shot framework.
• To leverage ADMM to enforce channel-level sparsity during training, enabling structured sparsity at the filter level.
• To evaluate the proposed method on standard benchmarks (MNIST, CIFAR-10) with widely used architectures (LeNet-5, AlexNet).
• To compare the performance of single-shot pruning against state-of-the-art iterative pruning methods under various compression ratios.
• To analyze the effectiveness of ADMM in inducing filter-level sparsity and improving generalization after pruning.

Proposed method

• Train a deep convolutional neural network (DCNN) using ADMM to enforce channel-level sparsity, where sparsity is enforced via an augmented Lagrangian formulation.
• Introduce a filter-level sparsity constraint using indicator functions $ g_i(W_i) $, where $ \text{card}(W_i) \leq l_i $, with $ l_i $ being the target number of remaining filters per layer.
• Apply a heuristic pruning criterion based on $ l_1 $-norm of filter weights to identify and remove the least important filters after ADMM training.
• Perform a single fine-tuning stage after pruning to recover performance degradation, using standard training procedures.
• Use the ADMM framework to iteratively update weights $ W $, dual variables $ Y $, and auxiliary variables $ Z $, enforcing the constraint $ W = Z $ through alternating minimization.
• Formulate the pruning problem as a constrained optimization: minimize the loss $ C(W,D) $ plus sparsity-inducing penalty $ \sum g_i(W_i) $, subject to filter count constraints.

Experimental results

Research questions

• RQ1 Can ADMM be effectively used to induce channel-level sparsity during training to enable single-shot pruning?
• RQ2 Does single-shot pruning with ADMM outperform existing iterative pruning methods in terms of accuracy and compression efficiency?
• RQ3 How does the $ l_1 $-norm of filters evolve during ADMM training, and does it correlate with filter importance?
• RQ4 What is the impact of ADMM on the generalization performance of pruned networks compared to standard training?
• RQ5 Can the proposed method maintain high accuracy at high pruning ratios without iterative refinement?

Key findings

• On AlexNet with CIFAR-10, the proposed method achieves 77.17% top-1 accuracy at 50% pruning ratio, outperforming Taylor expansion (TE) with 73.06% (no extra fine-tuning) and 75.47% (with extra fine-tuning).
• At 75% pruning ratio, the proposed method achieves 72.04% accuracy, significantly outperforming TE (62.72% without extra fine-tuning and 70.03% with).
• At 87.5% pruning ratio, the proposed method maintains 64.17% accuracy, compared to 51.83% for TE without extra fine-tuning and 60.92% with.
• ADMM training leads to a significant number of filters having $ l_1 $-norms close to zero, confirming effective channel-level sparsity induction.
• The distance $ |W - Z| $ decreases substantially in deeper layers during ADMM training, indicating that less important filters are effectively sparsified.
• Despite slightly lower accuracy in the pre-trained ADMM model, the pruned network fine-tuned from the ADMM-optimized model consistently achieves higher accuracy than the non-ADMM baseline across all stages.