[Paper Review] Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

Shaohui Lin, Rongrong Ji+6 more|arXiv (Cornell University)|Mar 22, 2019

Advanced Neural Network ApplicationsComputer Science61 references32 citations

TL;DR

The paper introduces Generative Adversarial Learning (GAL) to prune CNNs in an end-to-end, label-free manner by learning a sparse soft mask that jointly prunes channels, branches, and blocks, yielding strong compression and speedups.

ABSTRACT

Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of these structures by defining a new objective function with sparsity regularization to align the output of baseline and network with this mask. We then effectively solve the optimization problem by generative adversarial learning (GAL), which learns a sparse soft mask in a label-free and an end-to-end manner. By forcing more scaling factors in the soft mask to zero, the fast iterative shrinkage-thresholding algorithm (FISTA) can be leveraged to fast and reliably remove the corresponding structures. Extensive experiments demonstrate the effectiveness of GAL on different datasets, including MNIST, CIFAR-10 and ImageNet ILSVRC 2012. For example, on ImageNet ILSVRC 2012, the pruned ResNet-50 achieves 10.88\% Top-5 error and results in a factor of 3.7x speedup. This significantly outperforms state-of-the-art methods.

Motivation & Objective

• Motivate structured pruning that is efficient, slack, and label-free compared to multi-stage layer-wise methods.
• Propose a soft mask framework to sparsify and prune heterogeneous structures in CNNs.
• Develop an end-to-end GAL optimization leveraging a discriminator and FISTA to remove redundant structures.

Proposed method

• Introduce a sparse soft mask m that scales outputs of prunable structures (channels, branches, blocks).
• Formulate an adversarial objective with a generator (pruned network) and a discriminator to align pruned outputs with the baseline outputs (data loss via MSE).
• Solve the optimization with an Alternating GAN+FISTA approach: update discriminator via SGD, and prune via FISTA with L1 sparsity on m.
• Use an L1 sparsity penalty on m to enable structure removal when m_i → 0.
• Apply regularizers on weights and discriminator (L1/L2 or adversarial regularization) to balance the game.
• Employ FISTA to efficiently obtain exact-zero mask entries for pruning.

Experimental results

Research questions

• RQ1 Can a soft, learnable mask enable simultaneous pruning of heterogeneous CNN structures (channels, branches, blocks) in an end-to-end, label-free manner?
• RQ2 Does generative adversarial learning with an L1 sparse mask and FISTA yield superior compression and accuracy retention compared to traditional multi-stage pruning?
• RQ3 How does GAL perform across datasets (MNIST, CIFAR-10, ImageNet) and architectures (LeNet, VGG, DenseNet, GoogLeNet, ResNet, DenseNet-40, ResNet variants)?

Key findings

• On ImageNet, pruned ResNet-50 with GAL achieves 10.88% Top-5 error and 3.7× speedup.
• GAL demonstrates strong pruning efficiency across MNIST, CIFAR-10, and ImageNet with varying structures (channels, branches, blocks).
• Ablation shows adversarial regularization outperforms L1/L2 on the discriminator, improving pruning effectiveness.
• Joint pruning of blocks and channels on ResNet-50 (GAL-0.5-joint) yields higher speedup and compression than pruning blocks or channels alone.
• GAL often matches or surpasses state-of-the-art pruning methods across multiple networks and datasets.