[Paper Review] Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization

P. L. Yu, Joon Sern Lee+3 more|arXiv (Cornell University)|Jan 25, 2019

Advanced Bandit Algorithms Research49 references63 citations

TL;DR

The paper proposes a model-based deep RL architecture for dynamic portfolio optimization, introducing an Infused Prediction Module, a Data Augmentation Module with GANs, and a Behavior Cloning Module to stabilize training and improve risk-adjusted returns.

ABSTRACT

Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architecture with an autonomous trading agent such that, investment decisions and actions are made periodically, based on a global objective, with autonomy. In particular, without relying on a purely model-free RL agent, we train our trading agent using a novel RL architecture consisting of an infused prediction module (IPM), a generative adversarial data augmentation module (DAM) and a behavior cloning module (BCM). Our model-based approach works with both on-policy or off-policy RL algorithms. We further design the back-testing and execution engine which interact with the RL agent in real time. Using historical {\em real} financial market data, we simulate trading with practical constraints, and demonstrate that our proposed model is robust, profitable and risk-sensitive, as compared to baseline trading strategies and model-free RL agents from prior work.

Motivation & Objective

• Motivate the use of reinforcement learning for dynamic portfolio optimization under realistic trading constraints.
• Develop a model-based RL framework to address data efficiency, non-stationarity, and risk management in finance.
• Integrate prediction, data augmentation, and imitation components to enhance stability and performance of the trading agent.
• Evaluate the proposed architecture against baselines and model-free RL approaches using historical market data.

Proposed method

• Introduce an infused prediction module (IPM) that adds future observation predictions to the state used by RL algorithms.
• Incorporate a data augmentation module (DAM) using a recurrent GAN with maximum mean discrepancy (MMD) to generate realistic synthetic market data.
• Implement a behavior cloning module (BCM) that provides one-step greedy action demonstrations to constrain policy updates.
• Adopt a model-based adaptation of DDPG (and discuss applicability to PPO/TRPO) with an actor–critic setup.
• Extend the state with predictive features and market index signals, employing an LSTM-based or CNN-based feature extractor for the actor/critic networks.
• Train and test the agent on hourly-acted, daily-decided portfolios with transaction costs and slippage to reflect real-world constraints.

Experimental results

Research questions

• RQ1 Can a model-based RL framework with prediction, augmentation, and imitation components improve dynamic portfolio optimization under transaction costs and market frictions?
• RQ2 Do IPM, DAM, and BCM each contribute to improved risk-adjusted performance compared to model-free baselines and traditional strategies?
• RQ3 How does integrating future-based predictions and synthetic data affect stability and robustness in non-stationary financial environments?
• RQ4 Is the approach extensible to on-policy methods like PPO/TRPO beyond the off-policy DDPG setting?
• RQ5 What are the impacts on risk metrics such as drawdown and CVaR when employing the proposed modules?

Key findings

• The proposed architecture improves metrics such as Sharpe ratio, Sortino ratio, maximum drawdown, VaR, and CVaR relative to baselines and model-free RL agents.
• IPM provides significant performance gains by incorporating predicted future observations into the RL state.
• DAM helps reduce over-fitting and typically leads to portfolios with less volatility through synthetic data augmentation.
• BCM contributes to reducing portfolio weight volatility while preserving or enhancing returns in some cases.
• The framework demonstrates robustness and profitability under practical trading constraints and non-stationary market conditions.