[Paper Review] Optimizing Time Series Forecasting: A Comparative Study of Adam and Nesterov Accelerated Gradient on LSTM and GRU networks Using Stock Market data

Ayodeji Samuel Makinde|arXiv (Cornell University)|Sep 28, 2024

Stock Market Forecasting Methods5 citations

TL;DR

The paper compares Adam and Nesterov Accelerated Gradient (NAG) optimizers on LSTM and GRU models for stock price forecasting using Apple stock data; GRU with Adam yields the best RMSE and convergence.

ABSTRACT

Several studies have discussed the impact different optimization techniques in the context of time series forecasting across different Neural network architectures. This paper examines the effectiveness of Adam and Nesterov's Accelerated Gradient (NAG) optimization techniques on LSTM and GRU neural networks for time series prediction, specifically stock market time-series. Our study was done by training LSTM and GRU models with two different optimization techniques - Adam and Nesterov Accelerated Gradient (NAG), comparing and evaluating their performance on Apple Inc's closing price data over the last decade. The GRU model optimized with Adam produced the lowest RMSE, outperforming the other model-optimizer combinations in both accuracy and convergence speed. The GRU models with both optimizers outperformed the LSTM models, whilst the Adam optimizer outperformed the NAG optimizer for both model architectures. The results suggest that GRU models optimized with Adam are well-suited for practitioners in time-series prediction, more specifically stock price time series prediction producing accurate and computationally efficient models. The code for the experiments in this project can be found at https://github.com/AhmadMak/Time-Series-Optimization-Research Keywords: Time-series Forecasting, Neural Network, LSTM, GRU, Adam Optimizer, Nesterov Accelerated Gradient (NAG) Optimizer

Motivation & Objective

• Assess how Adam and Nesterov Accelerated Gradient optimizers perform on LSTM and GRU architectures for stock price time-series forecasting.
• Determine which model-optimizer combination provides the best predictive accuracy and convergence characteristics.
• Provide practical guidance on optimizer selection for time-series prediction in financial data.

Proposed method

• Train LSTM and GRU models using two optimizers (Adam and NAG) on Apple Inc. closing price data from 2014 to 2024.
• Preprocess data with normalization (Min-Max scaling), missing value imputation, and train/validation/test splits.
• Evaluate using training/validation losses, convergence behavior, and RMSE on test data.
• Use Keras (TensorFlow) on Google Colab with batch size 1, learning rate 0.001, and 1 epoch per run for initial comparisons.

Experimental results

Research questions

• RQ1 Which model (LSTM vs. GRU) yields lower RMSE for stock price forecasting with each optimizer?
• RQ2 Does Adam consistently outperform NAG across architectures in terms of convergence speed and stability?
• RQ3 How do training/validation losses evolve over epochs for the different model-optimizer combinations?
• RQ4 Are GRU models generally more effective than LSTM models for this stock market dataset when paired with Adam or NAG?

Key findings

• GRU with Adam achieves the lowest RMSE (172.4037).
• GRU models generally perform better than LSTM models in this study.
• Adam optimizer outperforms NAG for both LSTM and GRU in terms of RMSE and final losses.
• LSTM with Adam reduces training loss faster but can show instability; GRU with NAG shows stable convergence but higher RMSE than Adam variants.
• Across results, GRU-Adam is the strongest predictor for stock price forecasting in this dataset.