[论文解读] Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs

Tankut Can, Kamesh Krishnamurthy+1 more|arXiv (Cornell University)|Jan 31, 2020

Neural Networks and Reservoir Computing参考文献 37被引用 8

一句话总结

本文运用随机矩阵理论与平均场理论，分析GRU与LSTM中门控机制如何塑造循环神经网络的动力学特性。研究发现，GRU的更新门在特征值接近1处产生慢模式，从而实现长期记忆；而重置门则控制谱半径与相空间复杂度；LSTM的遗忘门同样促进慢模式的形成，所有门控机制均调节对训练稳定性与性能至关重要的谱特性。

ABSTRACT

Recurrent neural networks (RNNs) are powerful dynamical models for data with complex temporal structure. However, training RNNs has traditionally proved challenging due to exploding or vanishing of gradients. RNN models such as LSTMs and GRUs (and their variants) significantly mitigate these issues associated with training by introducing various types of gating units into the architecture. While these gates empirically improve performance, how the addition of gates influences the dynamics and trainability of GRUs and LSTMs is not well understood. Here, we take the perspective of studying randomly initialized LSTMs and GRUs as dynamical systems, and ask how the salient dynamical properties are shaped by the gates. We leverage tools from random matrix theory and mean-field theory to study the state-to-state Jacobians of GRUs and LSTMs. We show that the update gate in the GRU and the forget gate in the LSTM can lead to an accumulation of slow modes in the dynamics. Moreover, the GRU update gate can poise the system at a marginally stable point. The reset gate in the GRU and the output and input gates in the LSTM control the spectral radius of the Jacobian, and the GRU reset gate also modulates the complexity of the landscape of fixed-points. Furthermore, for the GRU we obtain a phase diagram describing the statistical properties of fixed-points. We also provide a preliminary comparison of training performance to the various dynamical regimes realized by varying hyperparameters. Looking to the future, we have introduced a powerful set of techniques which can be adapted to a broad class of RNNs, to study the influence of various architectural choices on dynamics, and potentially motivate the principled discovery of novel architectures.

研究动机与目标

• 理解GRU与LSTM中的架构门控机制如何影响随机初始化RNN的动力学特性。
• 解决当前对门控为何能提升RNN训练与性能的理论理解不足问题。
• 利用统计力学工具表征GRU与LSTM中状态转移雅可比矩阵的谱特性。
• 映射超参数（如门控权重）如何影响动力学模式与不动点复杂度。
• 通过将门控设计与动力学行为关联，为设计更优RNN架构提供理论基础。

提出的方法

• 应用随机矩阵理论（RMT）分析GRU与LSTM在初始化时的状态转移雅可比矩阵的谱特性。
• 使用平均场理论（MFT）计算在随机权重初始化下雅可比特征值的统计性质。
• 推导谱半径与特征值密度（尤其是接近1处）的解析表达式，以识别慢模式。
• 将雅可比矩阵建模为因循环连接而具有相关项的随机矩阵，从而与前馈网络相区分。
• 通过调节更新门与重置门权重（$a_z$, $a_r$）构建GRU的相图，识别不同动力学模式之间的转变。
• 在不同动力学模式下比较序列MNIST的训练性能，以验证理论预测。

实验结果

研究问题

• RQ1 GRU的更新门与重置门如何塑造状态转移雅可比矩阵的谱特性？
• RQ2 LSTM遗忘门在动力学中形成慢模式的作用是什么？
• RQ3 GRU与LSTM中的各类门如何控制谱半径与相空间复杂度？
• RQ4 能否为GRU推导出将超参数选择映射至动力学行为的相图？
• RQ5 理论上的动力学模式与实际在序列任务上的训练性能之间是否存在相关性？

主要发现

• GRU更新门在特征值接近1处引起雅可比矩阵特征值的聚集，形成慢模式，从而实现长期记忆与临界稳定性。
• GRU重置门控制谱半径，并调节相空间中不动点的拓扑复杂度。
• LSTM遗忘门主要负责特征值在接近1处的累积，有助于实现持久记忆。
• 所有门控机制在GRU与LSTM中均影响谱半径，但其影响程度与功能角色各不相同。
• 对于GRU，谱半径满足 $\rho({\bf J}_t) = \Theta(\sqrt{a_i})$（输入门），以及 $\rho({\bf J}_t) = \Theta(\sqrt{a_o})$（输出门）。
• 在序列MNIST上的实证训练表明，当 $a_h = 2.0$ 时，训练速度与准确率达到最优，此时零不动点变得不稳定——表明系统处于混沌边缘附近。