[Paper Review] Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction

Zhuangwei Shi, Hu Yang+2 more|arXiv (Cornell University)|Apr 6, 2022

Stock Market Forecasting Methods19 citations

TL;DR

This paper proposes an AttCLX hybrid model combining attention-based CNN-LSTM and XGBoost for stock price prediction. It uses ARIMA pre-processing, followed by a pretraining-fine-tuning framework where attention-based CNN-LSTM captures sequential features and XGBoost refines predictions, achieving state-of-the-art performance with RMSE of 0.01424 and R² of 0.88342 on Chinese stock data.

ABSTRACT

Stock market plays an important role in the economic development. Due to the complex volatility of the stock market, the research and prediction on the change of the stock price, can avoid the risk for the investors. The traditional time series model ARIMA can not describe the nonlinearity, and can not achieve satisfactory results in the stock prediction. As neural networks are with strong nonlinear generalization ability, this paper proposes an attention-based CNN-LSTM and XGBoost hybrid model to predict the stock price. The model constructed in this paper integrates the time series model, the Convolutional Neural Networks with Attention mechanism, the Long Short-Term Memory network, and XGBoost regressor in a non-linear relationship, and improves the prediction accuracy. The model can fully mine the historical information of the stock market in multiple periods. The stock data is first preprocessed through ARIMA. Then, the deep learning architecture formed in pretraining-finetuning framework is adopted. The pre-training model is the Attention-based CNN-LSTM model based on sequence-to-sequence framework. The model first uses convolution to extract the deep features of the original stock data, and then uses the Long Short-Term Memory networks to mine the long-term time series features. Finally, the XGBoost model is adopted for fine-tuning. The results show that the hybrid model is more effective and the prediction accuracy is relatively high, which can help investors or institutions to make decisions and achieve the purpose of expanding return and avoiding risk. Source code is available at https://github.com/zshicode/Attention-CLX-stock-prediction.

Motivation & Objective

• To address the limitations of traditional ARIMA models in capturing nonlinear stock price dynamics.
• To improve prediction accuracy by integrating deep learning with gradient-boosted trees in a hybrid architecture.
• To leverage attention mechanisms and sequence-to-sequence modeling for enhanced long-term dependency learning in financial time series.
• To validate the effectiveness of a pretraining-fine-tuning framework combining CNN-LSTM and XGBoost on real-world stock data.
• To provide a robust, decision-support tool for investors aiming to maximize returns and minimize risk in volatile markets.

Proposed method

• The model uses ARIMA(p=2, q=0, d=1) for initial preprocessing of stock price sequences to stabilize and remove trends.
• A pretraining phase employs an attention-based CNN-LSTM model within a sequence-to-sequence framework, where CNNs extract multi-scale deep features and BiLSTM decodes long-term temporal dependencies.
• The attention mechanism enables the model to focus on relevant historical patterns across long sequences, improving feature representation.
• Fine-tuning is performed using XGBoost regressor on the residual sequences from ARIMA, enhancing final prediction accuracy through ensemble learning.
• The hybrid architecture combines nonlinear modeling of deep neural networks with the robustness and interpretability of gradient-boosted trees.
• The entire framework follows a pretraining-fine-tuning paradigm, optimizing both feature extraction and final regression performance.

Experimental results

Research questions

• RQ1 Can an attention-based CNN-LSTM model effectively capture complex nonlinear patterns in stock price sequences compared to standard RNNs?
• RQ2 Does integrating XGBoost for fine-tuning residual sequences from ARIMA and deep learning models significantly improve prediction accuracy?
• RQ3 How does the proposed AttCLX model compare to state-of-the-art methods like ARIMA-NN, LSTM-KF, and Transformer-KF in terms of MAE, RMSE, and R² metrics?
• RQ4 To what extent does the pretraining-fine-tuning framework enhance feature learning and generalization in financial time series forecasting?
• RQ5 Can the hybrid model effectively reduce prediction error and increase R² while maintaining robustness on real-world stock data?

Key findings

• The AttCLX model achieved the lowest RMSE of 0.01424 and highest R² of 0.88342 among all compared models, outperforming ARIMA, ARIMA-NN, LSTM-KF, and Transformer-KF.
• The model's MAE of 0.01126 and MAPE of 0.01126 indicate high precision in predicting stock price movements on the Back of China (601988.SH) dataset.
• The pretraining-fine-tuning framework significantly improved performance, with XGBoost fine-tuning on residual sequences yielding better results than end-to-end training.
• The attention-based CNN-LSTM encoder-decoder architecture effectively modeled long-term dependencies and captured both local and global patterns in stock sequences.
• The AttCLX model outperformed even advanced models like BiLSTM-XGBoost and CNN-BiLSTM-XGBoost, demonstrating the superiority of attention-enhanced feature extraction.
• The loss curve of the AttCLX model (Fig. 16) showed stable convergence, indicating reliable training dynamics and robust generalization.