[Paper Review] Spatial-Winograd Pruning Enabling Sparse Winograd Convolution

Jiecao Yu, Jongsoo Park+1 more|arXiv (Cornell University)|Jan 8, 2019

Robotics and Sensor-Based Localization14 references9 citations

TL;DR

This paper proposes spatial-Winograd pruning, a two-stage method that first applies structured pruning in the spatial domain to preserve sparsity after Winograd transformation, then performs direct pruning and retraining in the Winograd domain using an importance factor matrix to accelerate convergence. It achieves 63%, 50%, and 74% Winograd-domain sparsity on CIFAR-10, CIFAR-100, and ImageNet with minimal accuracy loss.

ABSTRACT

Deep convolutional neural networks (CNNs) are deployed in various applications but demand immense computational requirements. Pruning techniques and Winograd convolution are two typical methods to reduce the CNN computation. However, they cannot be directly combined because Winograd transformation fills in the sparsity resulting from pruning. Li et al. (2017) propose sparse Winograd convolution in which weights are directly pruned in the Winograd domain, but this technique is not very practical because Winograd-domain retraining requires low learning rates and hence significantly longer training time. Besides, Liu et al. (2018) move the ReLU function into the Winograd domain, which can help increase the weight sparsity but requires changes in the network structure. To achieve a high Winograd-domain weight sparsity without changing network structures, we propose a new pruning method, spatial-Winograd pruning. As the first step, spatial-domain weights are pruned in a structured way, which efficiently transfers the spatial-domain sparsity into the Winograd domain and avoids Winograd-domain retraining. For the next step, we also perform pruning and retraining directly in the Winograd domain but propose to use an importance factor matrix to adjust weight importance and weight gradients. This adjustment makes it possible to effectively retrain the pruned Winograd-domain network without changing the network structure. For the three models on the datasets of CIFAR10, CIFAR-100, and ImageNet, our proposed method can achieve the Winograd domain sparsities of 63%, 50%, and 74%, respectively.

Motivation & Objective

• To address the incompatibility between pruning-induced sparsity and Winograd convolution, which typically destroys sparsity during transformation.
• To enable high sparsity in the Winograd domain without requiring network structure modifications or prohibitively slow retraining.
• To develop a method that preserves spatial-domain sparsity through structured pruning and enables effective Winograd-domain retraining.
• To reduce the need for extremely low learning rates in Winograd-domain retraining by introducing gradient adjustment via an importance factor matrix.

Proposed method

• Spatial structured pruning applies structured weight pruning in the spatial domain, designed to maximize sparsity transfer to the Winograd domain.
• The method uses predefined transformation matrices G and B to convert spatial-domain weights into the Winograd domain while preserving structural sparsity.
• Winograd direct pruning performs pruning and retraining directly in the Winograd domain, maintaining the original network architecture.
• An importance factor matrix F is introduced to adjust the gradient magnitude of Winograd-domain weights based on their impact on output activations.
• The importance factor matrix is computed using the sensitivity of output activations to weight changes, enabling more effective and faster retraining.
• The approach combines two stages: (1) spatial structured pruning to avoid Winograd-domain retraining initially, and (2) Winograd direct pruning with gradient adjustment to further increase sparsity.

Experimental results

Research questions

• RQ1 Can structured pruning in the spatial domain effectively transfer sparsity to the Winograd domain without requiring retraining in that domain?
• RQ2 Can direct pruning and retraining in the Winograd domain achieve high sparsity while maintaining model accuracy?
• RQ3 Can an importance factor matrix improve the convergence speed and effectiveness of Winograd-domain retraining without changing the network structure?
• RQ4 How does the proposed method compare to prior approaches like sparse Winograd convolution and Winograd-ReLU pruning in terms of sparsity and accuracy?

Key findings

• The proposed spatial-Winograd pruning achieves 63% Winograd-domain sparsity on CIFAR-10 with only 0.1% top-1 accuracy loss.
• On CIFAR-100, the method reaches 50% Winograd-domain sparsity while maintaining near-original accuracy.
• On ImageNet, the method achieves 74% Winograd-domain sparsity, surpassing prior methods, with less than 0.1% top-1 accuracy loss.
• Using the importance factor matrix reduces accuracy loss from 22% to 10% when pruning to 76% sparsity, demonstrating its effectiveness in preserving model performance.
• With 10 epochs of Winograd retraining, the importance factor matrix reduces accuracy loss to 0.2%, compared to 0.7% without gradient adjustment.
• The method avoids the need for extremely low learning rates in Winograd-domain retraining, enabling faster convergence than prior approaches.