[Paper Review] On the Origin of Deep Learning

Haohan Wang, Bhiksha Raj|arXiv (Cornell University)|Feb 24, 2017

Neural Networks and Applications131 references88 citations

TL;DR

This paper reviews the evolutionary history of deep learning models, tracing origins from associationism to modern architectures like CNNs, DBNs, and RNNs, and discusses how early ideas evolved into current deep learning forms.

ABSTRACT

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.

Motivation & Objective

• Trace the historical origins of deep learning from ancient associationism to modern neural networks.
• Analyze how early models were assembled into contemporary architectures (CNNs, DBNs, RNNs).
• Highlight the long, branching development paths and key milestones guiding current deep learning research.
• Provide a concise framework linking foundational ideas to present-day deep learning methods and identify future directions.

Proposed method

• Conduct a comprehensive literature and historical review of neural network milestones.
• Map evolutionary paths from early theories (associationism) to modern architectures.
• Synthesize how precursor ideas inform current model designs and optimization techniques.
• Compare with related reviews to contextualize the historical development of the field.

Experimental results

Research questions

• RQ1 What are the historical precursors and milestones that led to modern deep learning models?
• RQ2 How were early ideas assembled into architectures such as CNNs, DBNs, and RNNs?
• RQ3 What intuition and pathways connect ancient concepts to present deep learning practices?
• RQ4 What directions can guide future research in deep learning based on its history?

Key findings

• The paper traces a long historical lineage from associationism to contemporary deep learning architectures.
• It shows how CNNs, DBNs, and RNNs evolved from earlier concepts and were assembled into modern models.
• It argues that deep architectures emerged to address representation power and efficiency concerns.
• The work discusses optimization and training developments (e.g., backpropagation, dropout) as part of the evolutionary story.
• The paper provides directions to guide future research by understanding earlier milestones and their underlying ideas.