[Paper Review] Deep Reinforcement Learning in Portfolio Management

Zhipeng Liang, Kangkang Jiang+3 more|arXiv (Cornell University)|Aug 29, 2018

Advanced Bandit Algorithms Research17 references23 citations

TL;DR

This paper applies Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) — advanced continuous reinforcement learning algorithms — to portfolio management. It evaluates their performance across varied hyperparameters, objectives, markets, and features, offering actionable insights for hyperparameter tuning, feature selection, and data preparation in financial RL applications.

ABSTRACT

In this paper, we implement two state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) in portfolio management. Both of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rate, objective function, markets, feature combinations, in order to provide insights for parameter tuning, features selection and data preparation.

Motivation & Objective

• To investigate the applicability of state-of-the-art continuous reinforcement learning algorithms in portfolio management.
• To compare the performance of DDPG and PPO under diverse hyperparameter settings, objective functions, and market conditions.
• To provide practical guidance on hyperparameter tuning, feature selection, and data preparation for RL-based portfolio strategies.
• To assess the theoretical and empirical advantages of PPO in financial decision-making contexts.

Proposed method

• Employs Deep Deterministic Policy Gradient (DDPG), a continuous control algorithm, for end-to-end policy learning in portfolio allocation.
• Applies Proximal Policy Optimization (PPO), a stable and sample-efficient policy gradient method, to optimize portfolio decisions.
• Uses a continuous action space to represent asset weight allocations, enabling smooth and realistic portfolio rebalancing.
• Implements multiple objective functions and feature combinations to evaluate model robustness and adaptability.
• Trains and evaluates models across different financial markets to assess generalization and sensitivity to market regimes.
• Tunes hyperparameters such as learning rate, network architecture, and reward shaping to analyze their impact on performance.

Experimental results

Research questions

• RQ1 How do DDPG and PPO perform across different learning rates in portfolio management tasks?
• RQ2 What is the impact of varying objective functions on the stability and returns of RL-based portfolio strategies?
• RQ3 How do different feature combinations affect the performance of DDPG and PPO in portfolio optimization?
• RQ4 How do the models generalize across different financial markets and market regimes?
• RQ5 Which algorithm, DDPG or PPO, demonstrates superior performance and stability in portfolio management?

Key findings

• PPO demonstrates more stable training and better performance consistency across different hyperparameter settings compared to DDPG.
• The choice of objective function significantly influences the risk-adjusted returns of the portfolio strategies.
• Optimal performance is achieved with specific feature combinations that capture both trend and volatility dynamics.
• Learning rate tuning has a substantial impact on convergence speed and final policy quality.
• The models generalize across different markets, though performance varies with market regime and data quality.
• PPO's theoretical stability properties translate into practical advantages, reducing training instability and improving policy convergence.