[Paper Review] A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

Souvik Kundu, Saurav Prakash+3 more|arXiv (Cornell University)|Oct 2, 2019

Advanced Neural Network Applications29 references4 citations

TL;DR

This paper proposes pSConv, a pre-defined sparse kernel-based convolution for deep CNNs that fixes zero patterns in 2D filter kernels before training, enabling significant reductions in model complexity with minimal accuracy loss. It achieves up to 4.24× parameter reduction and 70% FLOP reduction on CIFAR-10 and Tiny ImageNet, outperforming ShuffleNet by 3–4% in accuracy under similar complexity constraints.

ABSTRACT

The high demand for computational and storage resources severely impede the deployment of deep convolutional neural networks (CNNs) in limited-resource devices. Recent CNN architectures have proposed reduced complexity versions (e.g. SuffleNet and MobileNet) but at the cost of modest decreases inaccuracy. This paper proposes pSConv, a pre-defined sparse 2D kernel-based convolution, which promises significant improvements in the trade-off between complexity and accuracy for both CNN training and inference. To explore the potential of this approach, we have experimented with two widely accepted datasets, CIFAR-10 and Tiny ImageNet, in sparse variants of both the ResNet18 and VGG16 architectures. Our approach shows a parameter count reduction of up to 4.24x with modest degradation in classification accuracy relative to that of standard CNNs. Our approach outperforms a popular variant of ShuffleNet using a variant of ResNet18 with pSConv having 3x3 kernels with only four of nine elements not fixed at zero. In particular, the parameter count is reduced by 1.7x for CIFAR-10 and 2.29x for Tiny ImageNet with an increased accuracy of ~4%.

Motivation & Objective

• To address the high computational and storage demands of deep CNNs in resource-constrained edge devices.
• To improve the accuracy-complexity trade-off in model compression by introducing a pre-defined sparsity mechanism in convolutional kernels.
• To enable efficient training and inference by structuring kernel sparsity in a way that simplifies hardware and software implementation.
• To demonstrate that pre-defined sparsity in CNNs can be as effective as in fully-connected layers, despite CNNs' inherent spatial sparsity.

Proposed method

• pSConv uses a fixed, pre-defined sparse pattern in 2D convolutional kernels, where only a subset of kernel weights (kernel support) are non-zero and remain fixed during training and inference.
• For each 3×3 kernel, the number of non-zero elements is reduced to 2, 4, or 9 (out of 9), with zero locations chosen pseudo-randomly and held constant.
• The method is applied to ResNet18 and VGG16 architectures on CIFAR-10 and Tiny ImageNet, replacing standard convolutions with pSConv layers.
• The approach is orthogonal to group-wise and point-wise convolutions, allowing for potential integration in future architecture search.
• Training is performed end-to-end with standard optimization, while sparsity is enforced via fixed kernel patterns rather than pruning.
• The method enables structured, predictable connectivity patterns, facilitating efficient software and hardware deployment.

Experimental results

Research questions

• RQ1 Can pre-defined sparse kernel patterns in 2D convolutions significantly reduce model complexity without degrading accuracy?
• RQ2 How does pSConv compare to state-of-the-art lightweight CNNs like ShuffleNet in terms of accuracy and parameter/FLOP reduction?
• RQ3 Does pre-defined sparsity in CNNs yield similar benefits as seen in fully-connected layers, despite CNNs' inherent spatial sparsity?
• RQ4 Can pSConv maintain training convergence and model performance when applied to deeper architectures like VGG16 and ResNet18?
• RQ5 What is the optimal kernel support size (number of non-zero elements) for balancing complexity reduction and accuracy in pSConv?

Key findings

• On CIFAR-10, a ResNet18 variant with pSConv and kernel support size 2 achieved 3.27× parameter reduction and 3.3% higher accuracy than ShuffleNet.
• On Tiny ImageNet, a pSConv-based ResNet18 with kernel support size 4 achieved 2.29× parameter reduction and ~3% higher accuracy than ShuffleNet.
• For VGG16 on CIFAR-10, pSConv reduced parameters by up to 34.0% and FLOPs by up to 75.4% with only a 1.4% drop in test accuracy.
• For VGG16 on Tiny ImageNet, pSConv reduced parameters by up to 28.1% and FLOPs by up to 78.4% with a 2.4% drop in test accuracy.
• Training convergence curves for pSConv models closely matched those of standard models, indicating stable optimization.
• The method enables up to 70% reduction in FLOPs with negligible accuracy degradation, demonstrating strong complexity-performance trade-offs.