[论文解读] A Critical Review of Recurrent Neural Networks for Sequence Learning

Zachary C. Lipton, John Berkowitz+1 more|arXiv (Cornell University)|May 29, 2015

Multimodal Machine Learning Applications参考文献 70被引用 2,087

一句话总结

一项对循环神经网络（RNN）在序列学习中的综述，涵盖架构（如 LSTM 和 BRNN）、训练挑战（梯度消失/爆炸）以及历史发展，强调比生物学可行性更关注实证结果。

ABSTRACT

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.

研究动机与目标

• 解释为何在实际任务中显式建模序列性是有价值的。
• 讨论马尔可夫模型和传统前馈网络在序列情境中的局限性。
• 提供一个连贯、独立的 RNN 架构、训练挑战与实际结果的综述。

提出的方法

• 回顾并综合三十年的 RNN 研究。
• 澄清符号表示并统一来自不同来源的术语。
• 解释 RNN 的前向与后向传播以及通过时间的反向传播的作用。

实验结果

研究问题

• RQ1 为什么显式的序列建模对于实际任务和长程依赖是必要的？
• RQ2 在处理时间依赖和长程上下文时，RNN 与马尔可夫模型有何比较？
• RQ3 哪些架构、训练技术和优化手段促成了大规模 RNN 学习的成功？
• RQ4 在 RNN 发展中有哪些历史里程碑和关键的经验发现？
• RQ5 现代 RNN 变体（如 LSTM 和 BRNN）如何解决训练挑战并提高性能？

主要发现

• RNNs 能捕捉超出固定上下文窗口的长程依赖，解决简单窗口和马尔可夫模型的局限。
• 诸如梯度消失与梯度爆炸的训练挑战促使了 LSTM 及相关架构的发展。
• 通过时间的反向传播使 RNN 能跨多个时间步进行端到端训练。
• BRNNs 与 LSTM 架构，以及优化和并行计算的进展，推动了序列任务的显著经验进展。
• 如神经图灵机（NTMs）及外部记忆等扩展进一步扩展了 RNN 的能力。