[论文解读] On the Origin of Deep Learning

Haohan Wang, Bhiksha Raj|arXiv (Cornell University)|Feb 24, 2017

Neural Networks and Applications参考文献 131被引用 88

一句话总结

本论文回顾深度学习模型的演化历史，追溯起源从关联主义到现代架构如卷积神经网络CNN、深度信念网络DBN、循环神经网络RNN，并讨论早期思想如何演变为当前深度学习形式。

ABSTRACT

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.

研究动机与目标

• 追溯深度学习从古代关联主义到现代神经网络的历史起源。
• 分析早期模型如何被组装成当代架构（CNN、DBN、RNN）。
• 突出漫长而曲折的发展路径及指引当前深度学习研究的关键里程碑。
• 提供一个简明框架，将基础思想与当今深度学习方法联系起来并识别未来方向。

提出的方法

• 进行对神经网络里程碑的全面文献与历史回顾。
• 绘制从早期理论（关联主义）到现代架构的进化路径。
• 综合前期思路如何影响当前模型设计与优化技术。
• 与相关评述进行比较，以将该领域的历史发展置于情境中。

实验结果

研究问题

• RQ1 导致现代深度学习模型的历史前驱和里程碑是什么？
• RQ2 早期思想如何被组装成如CNN、DBN和RNN等架构？
• RQ3 哪些直觉与路径将古老概念与当今深度学习实践连接起来？
• RQ4 基于历史，哪些方向可以为深度学习的未来研究提供指导？

主要发现

• 本文追溯了从关联主义到当代深度学习架构的漫长历史血脉。
• 它展示了CNN、DBN和RNN如何从早期概念演化并组装成现代模型。
• 它认为深度架构的出现是为了解决表示能力和效率方面的担忧。
• 该研究将优化与训练方面的发展（如反向传播、 dropout）视为演化故事的一部分。
• 本文通过理解早期里程碑及其潜在思想，为未来研究提供方向。