[论文解读] A journey in ESN and LSTM visualisations on a language task

Alexandre Variengien, Xavier Hinaut|arXiv (Cornell University)|Dec 3, 2020

Neural Networks and Reservoir Computing参考文献 48被引用 6

一句话总结

本文在跨情境语言学习任务上比较了回声状态网络（ESN）与长短期记忆网络（LSTM）的性能，采用一种基于UMAP降维的新型可视化技术——循环状态空间可视化（RSSviz）。结果表明，ESN在未经调优的情况下训练速度更快，且在更困难的数据集上表现优于LSTM；同时，两种模型均发展出聚焦于句子结构的相似内部动态。RSSviz揭示了LSTM中序列的分形状嵌入，并实现了对ESN学习过程及超参数影响的动态分析。

ABSTRACT

Echo States Networks (ESN) and Long-Short Term Memory networks (LSTM) are two popular architectures of Recurrent Neural Networks (RNN) to solve machine learning task involving sequential data. However, little have been done to compare their performances and their internal mechanisms on a common task. In this work, we trained ESNs and LSTMs on a Cross-Situationnal Learning (CSL) task. This task aims at modelling how infants learn language: they create associations between words and visual stimuli in order to extract meaning from words and sentences. The results are of three kinds: performance comparison, internal dynamics analyses and visualization of latent space. (1) We found that both models were able to successfully learn the task: the LSTM reached the lowest error for the basic corpus, but the ESN was quicker to train. Furthermore, the ESN was able to outperform LSTMs on datasets more challenging without any further tuning needed. (2) We also conducted an analysis of the internal units activations of LSTMs and ESNs. Despite the deep differences between both models (trained or fixed internal weights), we were able to uncover similar inner mechanisms: both put emphasis on the units encoding aspects of the sentence structure. (3) Moreover, we present Recurrent States Space Visualisations (RSSviz), a method to visualize the structure of latent state space of RNNs, based on dimension reduction (using UMAP). This technique enables us to observe a fractal embedding of sequences in the LSTM. RSSviz is also useful for the analysis of ESNs (i) to spot difficult examples and (ii) to generate animated plots showing the evolution of activations across learning stages. Finally, we explore qualitatively how the RSSviz could provide an intuitive visualisation to understand the influence of hyperparameters on the reservoir dynamics prior to ESN training.

研究动机与目标

• 比较ESN与LSTM在跨情境语言学习任务上的性能与内部动态。
• 探究尽管架构不同，ESN与LSTM是否发展出相似的内部表征。
• 开发并验证一种新型可视化方法RSSviz，用于分析循环状态空间动力学。
• 探索在训练前，池化超参数（谱半径与泄漏率）对ESN状态空间拓扑结构的影响。
• 实现对ESN与LSTM中学习进展、困难样本及模型行为的直观可视化分析。

提出的方法

• 在模拟婴儿词汇-意义习得的跨情境学习（CSL）数据集上训练ESN与LSTM。
• 应用UMAP进行非线性降维，以可视化两种模型的高维循环状态空间。
• 开发RSSviz，一种将池化状态通过UMAP映射并按输入或时间步着色的可视化框架。
• 利用RSSviz通过动画可视化分析ESN的学习阶段，展示训练过程中激活状态的演化。
• 使用训练前的RSSviz探索池化超参数（谱半径与泄漏率）对状态空间结构的影响。
• 通过内部单元激活分析，比较ESN与LSTM中功能角色的差异，重点关注句子结构编码。

实验结果

研究问题

• RQ1 ESN与LSTM在语言习得任务上是否达到可比性能，其训练动态有何差异？
• RQ2 尽管架构与训练机制不同，ESN与LSTM是否发展出相似的内部表征？
• RQ3 RSSviz能否揭示RNN潜在状态空间中如分形状嵌入等有意义的结构模式？
• RQ4 池化超参数（谱半径与泄漏率）在训练前如何影响池化状态空间的拓扑结构？
• RQ5 RSSviz能否用于检测困难样本，并可视化ESN中的学习进展？

主要发现

• LSTM在基础CSL语料上达到最低误差，但ESN训练速度显著更快，且在更困难的数据集上无需任何超参数调优即表现优于LSTM。
• 尽管ESN采用固定池化权重、LSTM采用可训练循环权重，两种模型均强调编码句子结构的单元，表明存在共享的功能机制。
• RSSviz揭示了LSTM潜在状态空间中序列的分形状嵌入，提示此类拓扑结构可能内生于RNN的序列处理过程。
• 对于ESN，RSSviz可识别困难训练样本，并支持生成动画可视化，显示学习呈阶段进行，单对象句子先于多对象句子被学习。
• RSSviz通过可视化谱半径与泄漏率对训练前状态空间结构的影响，为ESN池化中非线性与记忆之间的权衡提供了定性洞察。
• 本研究表明，RSSviz可加速超参数搜索，并通过训练前对池化质量与动态的可视化，指导新学习规则的设计。