[Paper Review] Exploring Sparsity in Recurrent Neural Networks

Sharan Narang, Erich Elsen+2 more|arXiv (Cornell University)|Apr 17, 2017

Advanced Neural Network Applications16 references145 citations

TL;DR

The paper presents a pruning-based method to progressively zero out weights during RNN training, achieving highly sparse models with maintained or improved accuracy and notable speedups.

ABSTRACT

Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.

Motivation & Objective

• Motivate reducing RNN parameter counts to enable deployment on mobile and embedded devices.
• Develop an in-training pruning method that yields sparse weight matrices without extra retraining.
• Demonstrate that pruning can maintain or improve accuracy while reducing model size.
• Quantify potential inference speedups from sparse recurrent layers and discuss deployment implications.

Proposed method

• Maintain a mask per weight and a monotonically increasing pruning threshold.
• Prune weights by zeroing parameters below a per-layer threshold updated periodically during training.
• Use layer-specific threshold functions governed by a small set of hyper-parameters (start_itr, ramp_itr, end_itr, theta, phi, freq).
• Prune recurrent and linear layers; do not prune biases or batch-norm parameters.
• Compare gradual pruning with hard pruning and with larger dense baselines to recover accuracy.
• Demonstrate applicability to GRU and vanilla RNN architectures within the Deep Speech 2 framework.

Experimental results

Research questions

• RQ1 Can in-training pruning of RNN weights achieve high sparsity with minimal accuracy loss?
• RQ2 How does gradual pruning compare to hard pruning in terms of final performance and parameter reduction?
• RQ3 What are the practical deployment benefits (memory, bandwidth, speed) of sparse RNNs on real hardware?

Key findings

• Achieve sparsity of around 88% to 92% in recurrent and linear layers after pruning.
• Pruned larger models (e.g., with 2560–3072 hidden units) can outperform or closely match dense baselines while using significantly fewer parameters.
• Gradual pruning outperforms hard pruning by about 7%–9% for comparable parameter counts.
• Sparse RNNs show substantial memory compression (Deep Speech 2: from 268 MB to ~32–64 MB; GRU: from 460 MB to ~50 MB).
• GEMM/SpMV benchmarks show speedups from 3x to 7x for recurrent layers at high sparsity, depending on layer size and whether GRU or RNN is used.
• Pruning reduces training time compared to some prior methods and can enable on-device deployment when combined with quantization.