[Paper Review] The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

Md Zahangir Alom, Tarek M. Taha+7 more|arXiv (Cornell University)|Mar 3, 2018

Generative Adversarial Networks and Image Synthesis76 references435 citations

TL;DR

A comprehensive survey summarizing deep learning approaches, architectures, variants, applications, frameworks, and benchmarks, tracing developments from AlexNet onward across multiple domains.

ABSTRACT

Deep learning has demonstrated tremendous success in variety of application domains in the past few years. This new field of machine learning has been growing rapidly and applied in most of the application domains with some new modalities of applications, which helps to open new opportunity. There are different methods have been proposed on different category of learning approaches, which includes supervised, semi-supervised and un-supervised learning. The experimental results show state-of-the-art performance of deep learning over traditional machine learning approaches in the field of Image Processing, Computer Vision, Speech Recognition, Machine Translation, Art, Medical imaging, Medical information processing, Robotics and control, Bio-informatics, Natural Language Processing (NLP), Cyber security, and many more. This report presents a brief survey on development of DL approaches, including Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) including Long Short Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). In addition, we have included recent development of proposed advanced variant DL techniques based on the mentioned DL approaches. Furthermore, DL approaches have explored and evaluated in different application domains are also included in this survey. We have also comprised recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys have published on Deep Learning in Neural Networks [1, 38] and a survey on RL [234]. However, those papers have not discussed the individual advanced techniques for training large scale deep learning models and the recently developed method of generative models [1].

Motivation & Objective

• Survey the development of deep learning methods including DNN, CNN, RNN (LSTM, GRU), Auto-Encoder, Deep Belief Network, GAN, and DRL.
• Summarize recent advances and variants of these techniques and their training methods.
• Review applications across image processing, computer vision, NLP, medicine, robotics, cybersecurity, and more.
• Provide an overview of frameworks, SDKs, and benchmark datasets used to implement and evaluate deep learning approaches.

Proposed method

• Review and categorize deep learning methods by architecture and learning paradigm.
• Summarize notable variants and training approaches for large-scale models.
• Map DL approaches to application domains and evaluate their impact.
• Discuss recent frameworks, SDKs, and benchmark datasets used in DL research.

Experimental results

Research questions

• RQ1 What are the major deep learning architectures and their variants developed since the advent of AlexNet?
• RQ2 How have these DL approaches been applied across different domains and tasks?
• RQ3 What frameworks, SDKs, and benchmarks have facilitated implementation and evaluation of DL methods?
• RQ4 What are the notable trends and advancements in training large-scale deep learning models and generative approaches?

Key findings

• DL methods including CNNs, RNNs, AE, DBN, GANs, and DRL have achieved state-of-the-art performance across multiple domains compared to traditional machine learning.
• The survey integrates recent developments and advanced variants of the mentioned DL approaches.
• Applications span image processing, computer vision, speech recognition, NLP, medical imaging, robotics, bioinformatics, cybersecurity, and more.
• The paper compiles frameworks, SDKs, and benchmark datasets used for implementing and evaluating DL approaches.