[Paper Review] Manifold Regularized Dynamic Network Pruning

Yehui Tang, Yunhe Wang+5 more|arXiv (Cornell University)|Mar 10, 2021

Advanced Neural Network Applications60 references4 citations

TL;DR

This paper proposes ManiDP, a manifold regularized dynamic network pruning method that adaptively prunes filters based on input-dependent complexity and inter-instance feature similarity. By aligning sparsity regularization with instance complexity and preserving similarity in pruned sub-networks, it achieves 55.3% FLOPs reduction in ResNet-34 with only 0.57% top-1 accuracy drop on ImageNet, outperforming state-of-the-art methods.

ABSTRACT

Neural network pruning is an essential approach for reducing the computational complexity of deep models so that they can be well deployed on resource-limited devices. Compared with conventional methods, the recently developed dynamic pruning methods determine redundant filters variant to each input instance which achieves higher acceleration. Most of the existing methods discover effective sub-networks for each instance independently and do not utilize the relationship between different inputs. To maximally excavate redundancy in the given network architecture, this paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks (dubbed as ManiDP). We first investigate the recognition complexity and feature similarity between images in the training set. Then, the manifold relationship between instances and the pruned sub-networks will be aligned in the training procedure. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost compared to the state-of-the-art methods. For example, our method can reduce 55.3% FLOPs of ResNet-34 with only 0.57% top-1 accuracy degradation on ImageNet.

Motivation & Objective

• To address the limitation of static and independent dynamic pruning methods that fail to exploit relationships between input instances.
• To maximize redundancy removal in deep networks by incorporating manifold structure—specifically, instance complexity and feature similarity—into the pruning process.
• To develop a dynamic pruning framework that adaptively allocates network capacity based on input-specific demands.
• To preserve the structural and semantic relationships between inputs in the pruned sub-networks for improved generalization and efficiency.

Proposed method

• The method first analyzes the recognition complexity of each training instance to assign adaptive sparsity penalties based on input difficulty.
• It computes feature similarity between instances using representations from the network to model inter-instance relationships.
• An adaptive sparsity loss with instance-specific penalty weights ensures that complex inputs receive more active filters, while simple inputs use fewer.
• A similarity regularization term aligns the similarity matrices of input features with those of the corresponding pruned sub-networks to maintain structural consistency.
• The overall training objective combines sparsity loss and similarity loss, optimized end-to-end with learnable hyperparameters.
• The pruned network generates instance-specific sub-networks at inference time, dynamically selecting channels based on input complexity and similarity.

Experimental results

Research questions

• RQ1 Can incorporating instance-specific complexity into dynamic pruning improve model efficiency without sacrificing accuracy?
• RQ2 How does preserving inter-instance feature similarity in pruned sub-networks affect generalization and computational cost?
• RQ3 What is the impact of adaptive sparsity regularization on balancing network capacity and redundancy removal across diverse inputs?
• RQ4 Can manifold-regularized pruning outperform existing static and dynamic pruning methods in terms of FLOPs reduction and accuracy retention?

Key findings

• The proposed ManiDP method reduces 55.3% FLOPs in ResNet-34 with only 0.57% top-1 accuracy degradation on ImageNet, demonstrating superior efficiency-accuracy trade-off.
• The method achieves a 73.87% reduction in FLOPs and 67.42% memory reduction in ResNet-56 while maintaining 93.29% test accuracy, a 0.41% drop from the original.
• Disabling the similarity loss component leads to a 1.43% accuracy drop compared to 0.98% when preserved, validating its importance in maintaining performance.
• The method is robust to hyperparameter settings, performing well across a wide range of sparsity and similarity loss weights (e.g., λ′ ∈ [0.001, 0.01], γ ≈ 10.0).
• Sub-networks exhibit variable FLOPs based on input complexity: simpler images use compact sub-networks, while complex or ambiguous images activate more channels.
• Visualization confirms that sub-networks with lower FLOPs correctly handle clear, simple images (e.g., distinct birds or dogs), while higher FLOP sub-networks are used for ambiguous or complex inputs (e.g., small or large objects).