[Paper Review] A Novel Deep Reinforcement Learning Based Automated Stock Trading System Using Cascaded LSTM Networks

Jie Zou, Jiashu Lou+2 more|arXiv (Cornell University)|Dec 6, 2022

Stock Market Forecasting Methods4 citations

TL;DR

This paper proposes a cascaded LSTM-regularized Proximal Policy Optimization (CLSTM-PPO) model for automated stock trading, leveraging stacked LSTMs to extract temporal features from financial time series before policy optimization. The model achieves 5–52% higher cumulative returns and 8–52% better maximum profit rates than baselines, with performance gains most pronounced in the Chinese market, where cumulative returns improved by 84.4% and Sharpe ratio by 37.4%.

ABSTRACT

More and more stock trading strategies are constructed using deep reinforcement learning (DRL) algorithms, but DRL methods originally widely used in the gaming community are not directly adaptable to financial data with low signal-to-noise ratios and unevenness, and thus suffer from performance shortcomings. In this paper, to capture the hidden information, we propose a DRL based stock trading system using cascaded LSTM, which first uses LSTM to extract the time-series features from stock daily data, and then the features extracted are fed to the agent for training, while the strategy functions in reinforcement learning also use another LSTM for training. Experiments in DJI in the US market and SSE50 in the Chinese stock market show that our model outperforms previous baseline models in terms of cumulative returns and Sharp ratio, and this advantage is more significant in the Chinese stock market, a merging market. It indicates that our proposed method is a promising way to build a automated stock trading system.

Motivation & Objective

• Address the challenge of low signal-to-noise ratios and non-stationarity in financial time series, which hinder direct application of deep reinforcement learning (DRL) in stock trading.
• Overcome limitations of supervised learning methods in financial prediction, such as overfitting and reliance on labeled market states.
• Develop a DRL-based automated trading system that dynamically learns optimal buy/sell actions without requiring pre-labeled market direction labels.
• Enhance model generalization and profitability by integrating time-series feature extraction via cascaded LSTMs into the DRL framework.
• Evaluate the model across diverse markets (US, China, India, UK) to assess robustness and performance in both developed and emerging equity markets.

Proposed method

• Employ a cascaded LSTM (CLSTM) architecture to extract hierarchical temporal features from daily stock data, including price, volume, and technical indicators (MACD, RSI, CCI, ADX).
• Use the extracted features as the initial state representation for a Proximal Policy Optimization (PPO) agent in a Markov Decision Process (MDP) framework.
• Formulate the trading task as an MDP with states defined by adjusted prices, cash balance, share holdings, and technical indicators.
• Train the PPO agent using a reward function based on portfolio return, with potential improvements via Sharpe ratio normalization to balance risk and return.
• Implement a multi-agent ensemble strategy as a baseline for comparison, ensuring consistent training environments and evaluation metrics.
• Use a 7-year minimum historical data window to ensure sufficient training stability, especially for long-horizon DRL policy learning.

Experimental results

Research questions

• RQ1 Can a cascaded LSTM architecture effectively extract meaningful temporal patterns from noisy, non-stationary financial time series to improve DRL-based trading policy learning?
• RQ2 How does the CLSTM-PPO model compare to existing DRL and ensemble-based trading strategies in terms of cumulative returns, risk-adjusted performance, and profitability per trade across multiple global markets?
• RQ3 Does the model’s performance advantage vary significantly across developed and emerging markets, particularly in high-volatility environments like China’s A-share market?
• RQ4 To what extent does the integration of LSTM-based feature extraction enhance the agent’s ability to generalize and avoid overfitting in reinforcement learning for automated trading?
• RQ5 Can reward function normalization (e.g., using Sharpe ratio) improve training stability and reduce maximum drawdown without sacrificing return potential?

Key findings

• The CLSTM-PPO model outperforms baseline models in cumulative returns by 5% to 52% across all tested markets, with the most significant improvement in the Chinese market, where cumulative returns rose by 84.4%.
• Maximum earning rate improved by 8% to 52% compared to baselines, indicating stronger profit capture capability, especially in trending markets.
• Average profitability per trade increased by 6% to 14%, demonstrating enhanced decision quality in individual trades.
• The Sharpe ratio improved by 37.4% in the Chinese market compared to the ensemble baseline, indicating better risk-adjusted performance in emerging markets.
• The model achieved first place in both mean equity return (MER) and average profit per trade (APPT) across all four markets, confirming the effectiveness of cascaded LSTMs in feature extraction.
• Performance was limited in the UK market due to insufficient training data (less than 7 years), suggesting that long-horizon DRL training requires substantial historical data for optimal convergence.