[论文解读] Practical Deep Reinforcement Learning Approach for Stock Trading

Xiaoyang Liu, Zhuoran Xiong+3 more|arXiv (Cornell University)|Nov 19, 2018

Stock Market Forecasting Methods被引用 97

一句话总结

作者将 Deep Deterministic Policy Gradient (DDPG) 应用于一个包含 30 只股票的交易环境，在夏普比率和累积回报上超过道琼斯工业平均指数(DJIA)和最小方差组合。

ABSTRACT

Stock trading strategy plays a crucial role in investment companies. However, it is challenging to obtain optimal strategy in the complex and dynamic stock market. We explore the potential of deep reinforcement learning to optimize stock trading strategy and thus maximize investment return. 30 stocks are selected as our trading stocks and their daily prices are used as the training and trading market environment. We train a deep reinforcement learning agent and obtain an adaptive trading strategy. The agent's performance is evaluated and compared with Dow Jones Industrial Average and the traditional min-variance portfolio allocation strategy. The proposed deep reinforcement learning approach is shown to outperform the two baselines in terms of both the Sharpe ratio and cumulative returns.

研究动机与目标

• Motivate and formulate stock trading as a Markov Decision Process (MDP) to maximize investment return.
• Propose a deep reinforcement learning approach (DDPG) to handle large state and action spaces in trading.
• Demonstrate adaptive trading performance on historical stock data and compare against baselines.
• Assess profitability and risk via multiple financial metrics (return, volatility, Sharpe ratio).

提出的方法

• Model stock trading as an MDP with state s = [p, h, b], action a across D stocks, and reward r(s,a,s').
• Use Deep Deterministic Policy Gradient (DDPG) with an actor-critic architecture to map states to actions.
• Incorporate an experience replay buffer and target networks to stabilize training and decorrelate samples.
• Train and validate on historical data (30 DJIA stocks) from 2009–2018, with a training/validation/trading workflow.
• Evaluate performance using final portfolio value, annualized return, annualized std. error, and Sharpe ratio.
• Provide a detailed Algorithm 1 describing the DDPG training loop with update rules for Q and μ.

实验结果

研究问题

• RQ1 Does the DDPG-based trading strategy outperform the Dow Jones Industrial Average and the min-variance portfolio on historical data?
• RQ2 What are the gains in return and risk-adjusted performance (Sharpe ratio) when using DDPG for stock trading?
• RQ3 How does the proposed method perform across training, validation, and live-trading-like phases on 30 stocks?

主要发现

DDPG（我们）	Min-Variance	DJIA
Initial Portfolio Value	10,000	10,000	10,000
Final Portfolio Value	19,791	14,369	15,428
Annualized Return	25.87%	15.93%	16.40%
Annualized Std. Error	13.62%	9.97%	11.70%
Sharpe Ratio	1.79	1.45	1.27

• DDPG achieves a higher final portfolio value (19,791) than both the Min-Variance (14,369) and DJIA (15,428).
• DDPG shows a higher annualized return (25.87%) compared to Min-Variance (15.93%) and DJIA (16.40%).
• DDPG has a higher annualized standard error (13.62%) than Min-Variance (9.97%) and DJIA (11.70%).
• DDPG attains a Sharpe ratio of 1.79 versus 1.45 (Min-Variance) and 1.27 (DJIA).
• The results indicate the DDPG-based trading strategy can outperform benchmarks in both return and risk balance.