[论文解读] FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network

Aditya Kusupati, Manish Singh+4 more|arXiv (Cornell University)|Jan 8, 2019

Neural Networks and Applications参考文献 47被引用 97

一句话总结

本文介绍 FastRNN 和 FastGRNN，这些架构在稳定训练、显著减少模型尺寸的同时保持准确性，使在资源受限设备上的边缘推理能够实现千字节级的 RNN。

ABSTRACT

This paper develops the FastRNN and FastGRNN algorithms to address the twin RNN limitations of inaccurate training and inefficient prediction. Previous approaches have improved accuracy at the expense of prediction costs making them infeasible for resource-constrained and real-time applications. Unitary RNNs have increased accuracy somewhat by restricting the range of the state transition matrix's singular values but have also increased the model size as they require a larger number of hidden units to make up for the loss in expressive power. Gated RNNs have obtained state-of-the-art accuracies by adding extra parameters thereby resulting in even larger models. FastRNN addresses these limitations by adding a residual connection that does not constrain the range of the singular values explicitly and has only two extra scalar parameters. FastGRNN then extends the residual connection to a gate by reusing the RNN matrices to match state-of-the-art gated RNN accuracies but with a 2-4x smaller model. Enforcing FastGRNN's matrices to be low-rank, sparse and quantized resulted in accurate models that could be up to 35x smaller than leading gated and unitary RNNs. This allowed FastGRNN to accurately recognize the "Hey Cortana" wakeword with a 1 KB model and to be deployed on severely resource-constrained IoT microcontrollers too tiny to store other RNN models. FastGRNN's code is available at https://github.com/Microsoft/EdgeML/.

研究动机与目标

• 解决标准 RNN 的训练不稳定与不精确，以及门控/单位模型的高预测代价。
• 开发基于残差连接的 RNN（FastRNN），仅添加极少量额外参数以稳定训练。
• 通过引入一个门控机制，复用 RNN 矩阵，将 FastRNN 扩展为 FastGRNN，以在更小的模型尺寸下实现具有竞争力的准确性。
• 通过低秩、稀疏和量化实现极端模型压缩，同时尽量减少精度损失。

提出的方法

• 在 FastRNN 中引入带有两个可训练标量权重（α、β）的加权残差连接以稳定梯度。
• 推导理论界限，证明 FastRNN 的收敛与序列长度 T 无关，以及泛化界限也与 T 无关。
• 通过用逐坐标门 z_t 取代标量残差，并复用 RNN 矩阵，来实现 FastGRNN。
• 对 W 和 U 应用低秩、稀疏性和量化，以在保持准确性的同时获得千字节级模型（1–6 KB）。
• 用三阶段程序训练 FastGRNN：学习低秩表示、学习稀疏结构、使用固定支撑进行优化；然后应用字节量化实现整数-算术推理。

实验结果

研究问题

• RQ1 具备残差使能的 RNN（FastRNN）是否能够在与标准 RNN 相比的极少额外参数下实现稳定训练和具有竞争力的准确性？
• RQ2 通过门控扩展（FastGRNN）重用 RNN 矩阵，是否能够以更小的模型尺寸维持接近最先进门控 RNN 的准确性？
• RQ3 低秩、稀疏与量化表示在多大程度上可以压缩 FastGRNN 而不造成明显的精度损失？
• RQ4 千字节级的 FastGRNN 模型是否适用于实时设备端唤醒词检测及其他物联网任务？

主要发现

• FastRNN 因残差连接（带两个可学习权重）实现了可证明的稳定训练和比标准 RNN 更快的收敛。
• FastGRNN 在准确性上达到或超过最先进的门控 RNN，同时参数量比领先的门控/单位 RNN 少 2–4 倍。
• 在低秩、稀疏和量化约束下，FastGRNN 模型的大小可比领先的门控/单位 RNN 小最多 35 倍。
• 一个 1 KB 的 FastGRNN 模型能够以与更大模型相当的准确性识别 Hey Cortana 唤醒词，并可部署在资源极度受限的物联网设备上。
• 在物联网硬件上，FastGRNN 的预测速度明显更快（在某些开发板上快 18–42 倍）；在其他对比中快 25–45 倍。