[Paper Review] Synaptic Strength For Convolutional Neural Network

Lin Chen, Zhao Zhong+2 more|arXiv (Cornell University)|Nov 6, 2018

Advanced Memory and Neural Computing16 citations

TL;DR

This paper introduces Synaptic Strength, a novel parameter in convolutional neural networks that quantifies the importance of individual connections between feature maps. By normalizing weights and inputs and using this strength as a pruning criterion, the method achieves up to 96% connection pruning on CIFAR-10 with minimal accuracy drop, outperforming existing methods in both accuracy preservation and sparsity efficiency on ImageNet.

ABSTRACT

Convolutional Neural Networks(CNNs) are both computation and memory intensive which hindered their deployment in mobile devices. Inspired by the relevant concept in neural science literature, we propose Synaptic Pruning: a data-driven method to prune connections between input and output feature maps with a newly proposed class of parameters called Synaptic Strength. Synaptic Strength is designed to capture the importance of a connection based on the amount of information it transports. Experiment results show the effectiveness of our approach. On CIFAR-10, we prune connections for various CNN models with up to 96% , which results in significant size reduction and computation saving. Further evaluation on ImageNet demonstrates that synaptic pruning is able to discover efficient models which is competitive to state-of-the-art compact CNNs such as MobileNet-V2 and NasNet-Mobile. Our contribution is summarized as following: (1) We introduce Synaptic Strength, a new class of parameters for CNNs to indicate the importance of each connections. (2) Our approach can prune various CNNs with high compression without compromising accuracy. (3) Further investigation shows, the proposed Synaptic Strength is a better indicator for kernel pruning compared with the previous approach in both empirical result and theoretical analysis.

Motivation & Objective

• To address the high computational and memory demands of deep CNNs for mobile deployment.
• To develop a data-driven, end-to-end pruning method inspired by biological synaptic pruning.
• To introduce Synaptic Strength as a new parameter to measure connection importance in CNNs.
• To achieve high-accuracy, highly compressed CNNs through kernel-level sparsity.
• To evaluate the method on CIFAR-10 and ImageNet, demonstrating superiority over existing pruning techniques.

Proposed method

• Synaptic Strength is defined as the product of normalized input feature map activation and kernel weights, capturing information flow per connection.
• The method applies batch normalization and L2 normalization to both input features and kernel weights to stabilize and scale the synaptic strength.
• A sparsity regularization term (λ) is introduced to encourage pruning of low-importance connections during end-to-end training.
• Pruning is performed by zeroing out kernels corresponding to low synaptic strength values, resulting in kernel-level sparse models.
• The approach is compatible with Winograd convolution, preserving sparsity and enabling efficient inference on GPU.
• Ablation studies validate the necessity of both γ (batch norm scale) removal and kernel L2 normalization in the synaptic strength formulation.

Experimental results

Research questions

• RQ1 Can a data-driven, biologically inspired pruning method achieve higher compression rates than existing approaches?
• RQ2 Is Synaptic Strength a more accurate indicator of connection importance than existing methods like L1 norm or group LASSO?
• RQ3 Can kernel-level sparse CNNs be efficiently accelerated using Winograd convolution?
• RQ4 How does the proposed method perform in terms of accuracy and sparsity trade-off at high pruning rates?
• RQ5 Does the inclusion of normalization components (γ and kernel norm) significantly improve pruning performance?

Key findings

• On CIFAR-10, the method achieves up to 96% pruning of synaptic connections with no drop in accuracy, significantly outperforming existing methods at high sparsity levels.
• At 90% sparsity, the proposed method causes less than 1% accuracy drop, while SSL-based methods exceed 1% drop, indicating superior robustness.
• On ImageNet, the pruned ResNet-50 achieves 25.32% top-1 error with 5.9M parameters, matching or surpassing MobileNet-V2 and NasNet-Mobile in efficiency.
• The method enables 8× higher sparsity (4%) compared to Winograd-domain pruning (25%) while maintaining nearly identical accuracy (6.23% error vs. 7.40%).
• Ablation studies confirm that removing either the γ factor or kernel normalization degrades performance, validating the design of Synaptic Strength.
• The proposed method maintains high accuracy even at extreme pruning rates (e.g., 97.5%), demonstrating strong robustness and scalability.