[Paper Review] Learning Intrinsic Sparse Structures within Long Short-Term Memory

Wei Wen, Yuxiong He+7 more|arXiv (Cornell University)|Sep 15, 2017

Topic Modeling25 references103 citations

TL;DR

The paper introduces Intrinsic Sparse Structures (ISS) to compress LSTM and RHN models by learning grouped sparsity that reduces basic LSTM components while keeping the original architecture intact, achieving substantial speedups with little to no loss in perplexity on language modeling and QA tasks.

ABSTRACT

Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units. To overcome the problem, we propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency. By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures. Based on group Lasso regularization, our method achieves 10.59x speedup without losing any perplexity of a language modeling of Penn TreeBank dataset. It is also successfully evaluated through a compact model with only 2.69M weights for machine Question Answering of SQuAD dataset. Our approach is successfully extended to non- LSTM RNNs, like Recurrent Highway Networks (RHNs). Our source code is publicly available at https://github.com/wenwei202/iss-rnns

Motivation & Objective

• Motivate model compression for RNNs and enable efficient inference on devices and services.
• Define Intrinsic Sparse Structures (ISS) to maintain dimension consistency while reducing basic LSTM components.
• Develop a group Lasso-based learning method to remove ISS weight groups during training or fine-tuning.
• Demonstrate effectiveness on language modeling (Penn TreeBank) and machine reading comprehension (SQuAD) with LSTMs and RHNs.
• Show extensibility of ISS to RHNs and other RNN variants.

Proposed method

• Organize eight LSTM weight matrices into ISS weight groups corresponding to left/right connections of each ISS component.
• Encourage group sparsity by adding a group Lasso regularization term R(w)=sum over ISS groups of ||w_k^(n)||_2 to the loss (Equation 2).
• Update ISS weights with SGD including a regularization gradient term proportional to w_k/||w_k||_2 (Equation 3).
• Stabilize sparsity by thresholding small weights below a threshold tau per minibatch (Equation 4).
• Allow ISS to prune entire components (rows/columns) across connected layers to preserve dimension consistency and avoid invalid LSTM structures.
• Extend ISS learning from scratch or fine-tuning; apply to RHNs and BiDAF-based QA models.

Experimental results

Research questions

• RQ1 Can intrinsic sparse structures within LSTMs be learned to reduce the size of input updates, gates, hidden states, cell states, and outputs without breaking dimension consistency?
• RQ2 Does group Lasso-based ISS sparsity learning yield meaningful speedups and memory reductions while preserving or minimally impacting task performance?
• RQ3 How well does ISS apply to RHNs and state-of-the-art QA models like BiDAF in terms of compression and accuracy?
• RQ4 What are the trade-offs between sparsity level, perplexity/EM-F1 scores, and inference speed across languages and tasks?

Key findings

• Learning ISS reduces LSTM sizes from 1500 to 373 (first layer) and 315 (second layer) on Penn TreeBank while maintaining perplexity and achieving 10.59x inference speedup.
• A compact 25.2M-parameter design (vs 66.0M baseline) also preserves performance with 7.10x speedup, and direct design of smaller LSTMs underperforms compared to ISS learned from larger models.
• RHN experiments show width reduction from 830 to 517 (11.1M parameters) with no loss in perplexity, and down to 403 (7.6M) still competitive; larger reductions yield higher perplexity (up to 74.5/71.2).
• In SQuAD-based BiDAF, ISS sparsity reduces multiple LSTM modules while preserving EM/F1 within tolerable losses; training from scratch with ISS yields substantial compression (e.g., 2.69M baseline down to 0.88–2.29M in various modules) and speedups.
• ISS learning can outperform naive smaller designs trained from scratch, indicating the benefit of structure-aware sparsity in RNNs.