[论文解读] LARNN: Linear Attention Recurrent Neural Network

Guillaume Chevalier|arXiv (Cornell University)|Aug 16, 2018

Advanced Neural Network Applications参考文献 7被引用 8

一句话总结

LARNN 提出了一种新颖的循环神经网络架构，将多头自注意力机制整合到类似LSTM的单元中，利用过去单元状态的固定大小窗口，实现高效、注意力增强的记忆检索。该方法在基于传感器的人体活动识别任务上实现了91.924%的测试准确率，较原始LSTM高出0.271%，并发现：在注意力机制的线性层中应用批归一化ELU激活函数可进一步提升性能。

ABSTRACT

The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past cell state values for a limited window size $k$. The formulas are also derived from the Batch Normalized LSTM (BN-LSTM) cell and the Transformer Network for its Multi-Head Attention Mechanism. The Multi-Head Attention Mechanism is used inside the cell such that it can query its own $k$ past values with the attention window. This has the effect of augmenting the rank of the tensor with the attention mechanism, such that the cell can perform complex queries to question its previous inner memories, which should augment the long short-term effect of the memory. With a clever trick, the LARNN cell with attention can be easily used inside a loop on the cell state, just like how any other Recurrent Neural Network (RNN) cell can be looped linearly through time series. This is due to the fact that its state, which is looped upon throughout time steps within time series, stores the inner states in a "first in, first out" queue which contains the $k$ most recent states and on which it is easily possible to add static positional encoding when the queue is represented as a tensor. This neural architecture yields better results than the vanilla LSTM cells. It can obtain results of 91.92% for the test accuracy, compared to the previously attained 91.65% using vanilla LSTM cells. Note that this is not to compare to other research, where up to 93.35% is obtained, but costly using 18 LSTM cells rather than with 2 to 3 cells as analyzed here. Finally, an interesting discovery is made, such that adding activation within the multi-head attention mechanism's linear layers can yield better results in the context researched hereto.

研究动机与目标

• 为解决标准RNN在高效捕捉长距离依赖关系方面的局限性。
• 将注意力机制整合到循环架构中，以增强记忆召回能力，同时避免计算复杂度过度增加。
• 探究在序列建模任务中，对过去隐藏状态应用注意力机制是否能提升性能。
• 评估位置编码和残差连接等架构选择对模型性能的影响。
• 研究注意力机制中线性投影的激活函数是否能改善学习效果。

提出的方法

• LARNN单元维护一个固定大小的先进先出队列，存储最近的k个单元状态，实现有界内存访问。
• 对k个过去的单元状态应用多头自注意力机制，动态加权并聚合相关的历史信息。
• 注意力机制在单元内部应用，查询由当前输入生成，键和值来自过去状态，采用点积注意力和缩放点积计算。
• 在多头注意力机制的线性投影中使用批归一化和ELU激活函数，提升训练稳定性和性能。
• 模型在标准RNN循环中训练，每个时间步使用注意力增强的单元状态更新隐藏状态。
• 在堆叠的LARNN层之间应用残差连接，实现特征融合并改善梯度流动。

实验结果

研究问题

• RQ1 在不引入二次方复杂度的前提下，将局部注意力机制集成到过去单元状态中，是否能提升RNN的表征能力？
• RQ2 在多头注意力机制的线性层中应用批归一化激活函数，是否能提升序列建模任务中的泛化性能？
• RQ3 当在RNN中使用固定窗口注意力机制时，位置编码对性能有何影响？
• RQ4 RNN中轻量级注意力机制是否能在序列分类基准上超越标准LSTM？
• RQ5 对于最大性能而言，窗口大小k、注意力头数量和残差堆叠层数等超参数的最优配置是什么？

主要发现

• LARNN在基于传感器的人体活动识别数据集上达到91.924%的测试准确率，优于原始LSTM基线的91.653%。
• 在多头注意力机制的线性投影中使用批归一化ELU激活函数，可提升测试性能，表明该架构具有有益的归纳偏置。
• 以密集方式应用位置编码未提升性能，表明在此特定设置下其效用有限。
• 以残差方式堆叠LARNN单元，相比简单的顺序（逐层）堆叠配置，性能更优。
• 当k固定时，模型保持线性复杂度O(nk²)，为全连接自注意力提供了可扩展的替代方案，同时通过近期记忆的注意力机制仍能捕捉长距离依赖。
• 该架构表明，注意力机制可有效嵌入RNN中以增强记忆召回，支持RNN在序列建模中的持续相关性。