[Paper Review] Efficient Sparse-Winograd Convolutional Neural Networks

Xingyu Liu, Jeff Pool+2 more|arXiv (Cornell University)|Feb 18, 2018

Advanced Neural Network Applications16 references47 citations

TL;DR

The paper introduces a Winograd-ReLU CNN that trains and prunes weights in the Winograd domain and moves ReLU to the Winograd domain to align sparsity with computation, achieving up to about 10x fewer multiplications with minimal accuracy loss across CIFAR-10, CIFAR-100, and ImageNet.

ABSTRACT

Convolutional Neural Networks (CNNs) are computationally intensive, which limits their application on mobile devices. Their energy is dominated by the number of multiplies needed to perform the convolutions. Winograd's minimal filtering algorithm (Lavin, 2015) and network pruning (Han et al., 2015) can reduce the operation count, but these two methods cannot be directly combined $-$ applying the Winograd transform fills in the sparsity in both the weights and the activations. We propose two modifications to Winograd-based CNNs to enable these methods to exploit sparsity. First, we move the ReLU operation into the Winograd domain to increase the sparsity of the transformed activations. Second, we prune the weights in the Winograd domain to exploit static weight sparsity. For models on CIFAR-10, CIFAR-100 and ImageNet datasets, our method reduces the number of multiplications by $10.4 imes$, $6.8 imes$ and $10.8 imes$ respectively with loss of accuracy less than $0.1\%$, outperforming previous baselines by $2.0 imes$-$3.0 imes$. We also show that moving ReLU to the Winograd domain allows more aggressive pruning.

Motivation & Objective

• Motivate reducing CNN compute for mobile deployment by combining Winograd convolution with sparsity.
• Enable sparsity exploitation in both weights and activations when using Winograd transforms.
• Propose a Winograd-ReLU architecture with transform-domain pruning and training procedures.
• Demonstrate significant multiplication reductions on CIFAR-10, CIFAR-100, and ImageNet with negligible accuracy impact.

Proposed method

• Use Winograd’s minimal filtering with 4x4 transformed kernels (p=4) to reduce multiplies.
• Move ReLU from spatial domain to Winograd domain to increase activation sparsity at multiplication time.
• Prune the transformed Winograd-domain weights after transformation to exploit static weight sparsity.
• Train densely in the Winograd domain, prune transformed kernels with a fixed threshold to achieve target density, and retrain with a sparsity mask.
• Provide backpropagation updates for Winograd-weight gradients and input activations in the transformed domain (equation 5).

Experimental results

Research questions

• RQ1 Can sparsity be effectively exploited when using Winograd convolution by relocating ReLU to the Winograd domain?
• RQ2 Does pruning in the Winograd domain preserve accuracy while delivering large reductions in multiplications across standard benchmarks?
• RQ3 What are the comparative gains in computation when combining Winograd with sparse activations and weights across CIFAR-10, CIFAR-100, and ImageNet?

Key findings

• On CIFAR-10, the Winograd-ReLU model yields up to 10.4x reduction in multiplies with <0.1% accuracy loss compared to baselines.
• On CIFAR-100, the Winograd-ReLU model yields up to 10.8x reduction in multiplies with <0.1% accuracy loss.
• On ImageNet, the Winograd-ReLU model yields up to 10.8x reduction in multiplies with <0.1% accuracy loss.
• Across datasets, the Winograd-ReLU approach consistently outperforms traditional Winograd or spatial pruning baselines by 2.0x–3.0x in workload reduction.