[Paper Review] A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU

Farhad Mortezapour Shiri, Thinagaran Perumal+2 more|arXiv (Cornell University)|May 27, 2023

COVID-19 diagnosis using AI112 citations

TL;DR

A survey and empirical study comparing classic and modern deep learning models (CNN, RNN, LSTM, GRU, TCN, Transformer) across text and image tasks using IMDB, ARAS, and Fruit-360 datasets, including CNN architectures.

ABSTRACT

Deep learning (DL) has emerged as a powerful subset of machine learning (ML) and artificial intelligence (AI), outperforming traditional ML methods, especially in handling unstructured and large datasets. Its impact spans across various domains, including speech recognition, healthcare, autonomous vehicles, cybersecurity, predictive analytics, and more. However, the complexity and dynamic nature of real-world problems present challenges in designing effective deep learning models. Consequently, several deep learning models have been developed to address different problems and applications. In this article, we conduct a comprehensive survey of various deep learning models, including Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Temporal Convolutional Networks (TCN), Transformer, Kolmogorov-Arnold networks (KAN), Generative Models, Deep Reinforcement Learning (DRL), and Deep Transfer Learning. We examine the structure, applications, benefits, and limitations of each model. Furthermore, we perform an analysis using three publicly available datasets: IMDB, ARAS, and Fruit-360. We compared the performance of six renowned deep learning models: CNN, RNN, Long Short-Term Memory (LSTM), Bidirectional LSTM, Gated Recurrent Unit (GRU), and Bidirectional GRU alongside two newer models, TCN and Transformer, using the IMDB and ARAS datasets. Additionally, we evaluated the performance of eight CNN-based models, including VGG (Visual Geometry Group), Inception, ResNet (Residual Network), InceptionResNet, Xception (Extreme Inception), MobileNet, DenseNet (Dense Convolutional Network), and NASNet (Neural Architecture Search Network), for image classification tasks using the Fruit-360 dataset.

Motivation & Objective

• Survey the structure, applications, benefits, and limitations of major deep learning models.
• Conduct empirical comparisons of selected sequence models (CNN, RNN, LSTM, GRU, Bidirectional variants) and newer models (TCN, Transformer) on textual data (IMDB, ARAS).
• Evaluate CNN-based image classification architectures (VGG, Inception, ResNet, InceptionResNet, Xception, MobileNet, DenseNet, NASNet) on Fruit-360 dataset.

Proposed method

• Provide a comprehensive literature survey of deep learning models including CNN, RNN, LSTM, GRU, Temporal Convolutional Networks (TCN), Transformer, Kolmogorov-Arnold networks (KAN), Generative Models, Deep Reinforcement Learning (DRL), and Deep Transfer Learning.
• Perform empirical evaluation on three public datasets (IMDB, ARAS, Fruit-360).
• Compare six well-known models (CNN, RNN, LSTM, Bidirectional LSTM, GRU, Bidirectional GRU) alongside two newer models (TCN, Transformer).
• Evaluate eight CNN-based image architectures (VGG, Inception, ResNet, InceptionResNet, Xception, MobileNet, DenseNet, NASNet) on the Fruit-360 dataset.

Experimental results

Research questions

• RQ1 How do CNN, RNN, LSTM, GRU, TCN, and Transformer compare in performance on textual datasets (IMDB and ARAS)?
• RQ2 How do eight CNN architectures perform on the Fruit-360 image classification task?
• RQ3 What are the strengths and limitations of each model class across the surveyed domains?
• RQ4 What insights emerge regarding when to prefer traditional RNN-based models versus newer architectures like TCN and Transformer?

Key findings

• The paper surveys a broad set of deep learning models and their applications across text and image domains.
• The study includes empirical comparisons of multiple sequence models (including Bidirectional variants) and newer architectures (TCN, Transformer).
• Eight CNN-based architectures are evaluated on the Fruit-360 image classification task to assess architectural differences.
• The analysis covers structure, benefits, and limitations of each model type across datasets IMDB, ARAS, and Fruit-360.