[Paper Review] Adversarial Deep Reinforcement Learning in Portfolio Management

Zhipeng Liang, Hao Chen+3 more|arXiv (Cornell University)|Aug 29, 2018

Stock Market Forecasting Methods57 citations

TL;DR

The paper compares DDPG, PPO, and PG for portfolio management on China stock data, introduces adversarial training to improve training efficiency and performance, and shows Policy Gradient (PG) often outperforms the others with adversarial enhancements.

ABSTRACT

In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rates, objective functions, feature combinations, in order to provide insights for parameters tuning, features selection and data preparation. We also conduct intensive experiments in China Stock market and show that PG is more desirable in financial market than DDPG and PPO, although both of them are more advanced. What's more, we propose a so called Adversarial Training method and show that it can greatly improve the training efficiency and significantly promote average daily return and sharpe ratio in back test. Based on this new modification, our experiments results show that our agent based on Policy Gradient can outperform UCRP.

Motivation & Objective

• Motivate applying deep reinforcement learning to portfolio management under transaction costs and finite horizons.
• Systematically evaluate three continuous RL algorithms (DDPG, PPO, PG) under varied hyperparameters, data preprocessing, and feature sets.
• Propose Adversarial Training to improve training efficiency and risk sensitivity in a financial setting.
• Explore effects of data normalization, network structure, and feature combinations on policy performance.

Proposed method

• Model portfolio management as a Markov decision process with transaction costs and log-wealth reward.
• Implement and compare DDPG, PPO, and Policy Gradient with neural network function approximators.
• Introduce Adversarial Training by adding noise to market prices and adapting the policy gradient accordingly.
• Use Identical Independent Evaluators (IIE) network architecture to handle multiple assets with shared parameters.
• Experiment with China Stock market data, five assets per portfolio, and data normalization across periods.
• Evaluate learning rate, feature combinations, and risk-adjusted objective considerations.

Experimental results

Research questions

• RQ1 Which of the continuous reinforcement learning algorithms (DDPG, PPO, PG) yields the best performance in Chinese equity portfolios under transaction costs?
• RQ2 How do hyperparameters (learning rate, network depth), feature selections, and data preprocessing affect policy performance across these algorithms?
• RQ3 Does adversarial training improve training efficiency and financial performance (e.g., average daily return and Sharpe ratio) in backtesting?
• RQ4 What is the impact of using the Identical Independent Evaluators architecture and feature combinations on scalability and learning stability?
• RQ5 Can these reinforcement learning approaches outperform traditional benchmarks under realistic market frictions?

Key findings

• Policy Gradient (PG) tends to be more desirable in the analyzed financial market than DDPG and PPO.
• Adversarial Training substantially improves training efficiency and increases average daily return and Sharpe ratio in backtesting.
• Feature choice matters; combining closing price with high price yields better performance in their experiments.
• Learning rate affects critic loss and stability, with actor learning rate influencing training dynamics and critic generalization.
• Data normalization and the IIE network structure support scalable learning across multiple assets.
• PG with adversarial modification can outperform certain benchmarks like UCRP under the study setup.