[论文解读] Intrinsically Sparse Long Short-Term Memory Networks

Shiwei Liu, Decebal Constantin Mocanu+1 more|arXiv (Cornell University)|Jan 26, 2019

Topic Modeling参考文献 19被引用 6

一句话总结

本文提出SET-LSTM，一种通过稀疏进化训练（SET）训练的稀疏连接长短期记忆网络，在情感分析任务上实现了最先进性能，参数量不足全连接LSTM的4%。通过在LSTM单元和嵌入层中初始化稀疏连接结构，并在训练过程中演化网络拓扑，SET-LSTM在降低内存和计算成本的同时，提升了或匹配了在四个数据集上的准确率。

ABSTRACT

Long Short-Term Memory (LSTM) has achieved state-of-the-art performances on a wide range of tasks. Its outstanding performance is guaranteed by the long-term memory ability which matches the sequential data perfectly and the gating structure controlling the information flow. However, LSTMs are prone to be memory-bandwidth limited in realistic applications and need an unbearable period of training and inference time as the model size is ever-increasing. To tackle this problem, various efficient model compression methods have been proposed. Most of them need a big and expensive pre-trained model which is a nightmare for resource-limited devices where the memory budget is strictly limited. To remedy this situation, in this paper, we incorporate the Sparse Evolutionary Training (SET) procedure into LSTM, proposing a novel model dubbed SET-LSTM. Rather than starting with a fully-connected architecture, SET-LSTM has a sparse topology and dramatically fewer parameters in both phases, training and inference. Considering the specific architecture of LSTMs, we replace the LSTM cells and embedding layers with sparse structures and further on, use an evolutionary strategy to adapt the sparse connectivity to the data. Additionally, we find that SET-LSTM can provide many different good combinations of sparse connectivity to substitute the overparameterized optimization problem of dense neural networks. Evaluated on four sentiment analysis classification datasets, the results demonstrate that our proposed model is able to achieve usually better performance than its fully connected counterpart while having less than 4\% of its parameters.

研究动机与目标

• 解决大型LSTM模型在资源受限环境下的内存与计算效率低下问题。
• 克服现有压缩技术对昂贵预训练全连接模型的依赖。
• 通过从初始架构设计引入稀疏性，实现高效训练与推理。
• 探究自适应稀疏连接结构是否能提供优于过度参数化密集网络的优化景观。
• 评估稀疏LSTM在真实世界序列NLP任务中的性能与鲁棒性。

提出的方法

• 提出SET-LSTM，一种新型LSTM架构，从初始阶段即在LSTM单元和嵌入层中采用稀疏连接结构。
• 应用稀疏进化训练（SET）过程，在训练期间演化稀疏拓扑结构，起始于Erdős-Rényi随机图结构。
• 用稀疏、可训练的自适应连接模式替代LSTM门中标准的全连接线性变换。
• 利用进化策略基于性能反馈迭代重连结构，同时优化权重与拓扑结构。
• 采用重连率超参数控制每轮训练的稀疏度演化，实现对数据的动态适应。
• 利用稀疏数据结构在训练与推理过程中降低内存带宽与计算开销。

实验结果

研究问题

• RQ1 能否从初始阶段即在LSTM中引入稀疏连接，以极低参数量实现高性能？
• RQ2 通过SET实现的自适应稀疏拓扑演化是否能带来优于密集、过度参数化LSTM的泛化能力？
• RQ3 SET-LSTM在多个情感分类基准上的性能与全连接LSTM相比如何？
• RQ4 固定拓扑结构与学习到的拓扑结构，以及权重初始化方式，对稀疏网络性能有何影响？
• RQ5 稀疏度与模型准确率之间是否存在权衡？是否存在实现有效性能的最优稀疏范围？

主要发现

• SET-LSTM在四个情感分析数据集中的三个上达到更高或相当的准确率，参数量最多减少25倍。
• 在IMDb数据集上，SET-LSTM仅使用全连接基线3.8%的参数量，达到80.14%的准确率，比其高出0.89%。
• 在Yelp 2018数据集上，SET-LSTM使用比全连接对应模型少96%的参数量，达到68.13%的准确率。
• Twitter数据的最优重连率为ζ = 0.9，每轮仅保留10%的连接，准确率达到79.37%。
• 使用随机初始化训练的固定拓扑结构准确率显著偏低（77.97%），远低于原始进化训练方法（78.89%），凸显了权重与拓扑联合优化的重要性。
• 即使在极端稀疏度（剪枝99.1%）下，SET-LSTM仍保持78.75%的准确率，超过全连接LSTM的77.19%，表明其对高稀疏度具有强鲁棒性。