[Paper Review] Block-Sparse Recurrent Neural Networks

Sharan Narang, Eric Undersander+1 more|arXiv (Cornell University)|Nov 8, 2017

Advanced Neural Network Applications22 references96 citations

TL;DR

The paper introduces block pruning and group lasso regularization to induce block sparsity in RNNs and GRUs, achieving 80–90% sparsity with minimal accuracy loss and about 10x model size reduction, applicable up to 32x32 blocks.

ABSTRACT

Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, we demonstrate that we can create block-sparse RNNs with sparsity ranging from 80% to 90% with small loss in accuracy. This allows us to reduce the model size by roughly 10x. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32x32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

Motivation & Objective

• Motivate reducing compute and memory in RNNs for speech recognition and related tasks.
• Propose block pruning and group lasso regularization to create block-structured sparsity in RNN/GRU matrices.
• Evaluate sparsity-accuracy tradeoffs on large-scale speech datasets with 4x4 to 32x32 blocks.
• Demonstrate hardware efficiency benefits of block-sparse formats over unstructured sparsity.

Proposed method

• Extend weight pruning to prune blocks by selecting a representative maximum magnitude per block and zeroing blocks below a threshold.
• Use a monotonically increasing pruning threshold with schedule parameters to reach high sparsity during training.
• Introduce group lasso regularization over weight blocks to encourage block-level sparsity, combined with pruning (GLP).
• Combine block pruning with group lasso (GLP) and analyze effects on accuracy and sparsity.
• Explore block sizes up to 32x32 and discuss hardware implications and memory overhead reductions.

Experimental results

Research questions

• RQ1 Can block pruning and group lasso regularization produce high block sparsity (80–90%) in RNNs/GRUs with minimal accuracy loss?
• RQ2 How do block size and pruning schedule affect accuracy, parameter count, and hardware efficiency on large speech datasets?
• RQ3 What is the comparative performance of block pruning, group lasso, and their combination (GLP) in RNN/GRU models?
• RQ4 How does increasing the dense model size prior to pruning influence final sparsity and accuracy?

Key findings

• Block pruning and group lasso with pruning achieve about 89–90% sparsity with modest CER degradation (8.8% for GRU, 16.7–16.0% for RNN depending on configuration).
• Sparse RNN/GRU models reduce parameters by roughly 10x, with GRU sparse 2560/3584 achieving CER near the dense baselines.
• Larger block sizes (up to 32x32) tolerate lower sparsity to maintain accuracy, while still benefiting from reduced memory overhead.
• Group lasso without pruning underperforms compared to GLP, indicating pruning aids sparsity effectiveness.
• Pruning larger, denser models (e.g., 2560/3072) can recover accuracy closer to dense baselines while maintaining high sparsity.