[Paper Review] Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution

Rundong Wang, Hongxin Wei+3 more|arXiv (Cornell University)|Dec 23, 2020

Advanced Bandit Algorithms Research19 references4 citations

TL;DR

This paper proposes HRPM, a hierarchical reinforcement learning framework that separates portfolio management from order execution to optimize long-term returns while minimizing trading costs like price slippage. By training a high-level policy for rebalancing and a low-level policy for execution within time windows, HRPM achieves significant performance gains over state-of-the-art methods in both U.S. and Chinese markets, with a 48.7% annualized return and superior risk-adjusted metrics.

ABSTRACT

Portfolio management via reinforcement learning is at the forefront of fintech research, which explores how to optimally reallocate a fund into different financial assets over the long term by trial-and-error. Existing methods are impractical since they usually assume each reallocation can be finished immediately and thus ignoring the price slippage as part of the trading cost. To address these issues, we propose a hierarchical reinforced stock trading system for portfolio management (HRPM). Concretely, we decompose the trading process into a hierarchy of portfolio management over trade execution and train the corresponding policies. The high-level policy gives portfolio weights at a lower frequency to maximize the long term profit and invokes the low-level policy to sell or buy the corresponding shares within a short time window at a higher frequency to minimize the trading cost. We train two levels of policies via pre-training scheme and iterative training scheme for data efficiency. Extensive experimental results in the U.S. market and the China market demonstrate that HRPM achieves significant improvement against many state-of-the-art approaches.

Motivation & Objective

• To address the limitation of existing reinforcement learning methods in portfolio management that ignore price slippage and assume instantaneous execution.
• To model the real-world hierarchy of portfolio managers and traders by structuring a two-level decision process: long-term rebalancing and short-term execution.
• To improve data efficiency and performance by pre-training low-level policies and using iterative training with entropy regularization.
• To empirically validate the framework's superiority in diverse market conditions, including high-volatility environments.

Proposed method

• HRPM formulates portfolio management with trading costs as a hierarchical Markov Decision Process (HMDP), separating long-term portfolio weight decisions from short-term execution actions.
• The high-level policy operates at a lower frequency to determine optimal portfolio weights and triggers the low-level policy for execution.
• The low-level policy uses action branching to handle multi-dimensional actions (buy/sell quantities and prices) and is trained via a pre-training and iterative fine-tuning scheme.
• An entropy bonus is introduced in the high-level policy to encourage portfolio diversification and reduce risk exposure.
• The framework leverages limit order book data and integrates real trading costs, including slippage, into the reward function.
• Training is conducted using historical price and limit order data from both the U.S. and China stock markets to ensure real-world applicability.

Experimental results

Research questions

• RQ1 Can a hierarchical reinforcement learning framework effectively decouple long-term portfolio optimization from short-term order execution to reduce trading costs?
• RQ2 Is price slippage a non-negligible component of trading cost that must be explicitly modeled for realistic performance?
• RQ3 Does incorporating an entropy bonus in the high-level policy improve risk diversification and overall portfolio stability?
• RQ4 How does HRPM compare to state-of-the-art methods in terms of return, risk-adjusted performance, and robustness across different market regimes?
• RQ5 To what extent does pre-training and iterative fine-tuning of low-level policies enhance data efficiency and overall system performance?

Key findings

• HRPM achieved an annualized return of 48.742% in the China market, outperforming the next best baseline (DPM) by 13% and the market index (SSE50) by 42%.
• In the U.S. market, HRPM recorded an annualized return of 27.089% with an entropy bonus coefficient of η=0.05, which was the highest among all tested configurations.
• HRPM demonstrated superior risk control, achieving the lowest maximum drawdown (MDD) of 0.117 in the U.S. market and maintaining strong performance in volatile conditions.
• The ablation study confirmed that commission fees alone are insufficient to capture total trading costs, with slippage being a dominant component, especially in large trades.
• The entropy bonus significantly improved risk-adjusted returns, with η=0.05 yielding the highest ARR and η=0.1 producing the lowest MDD, indicating effective risk diversification.
• Visual analysis of low-level policy behavior showed that it strategically sells at price peaks and buys at local troughs, minimizing adverse price impact and improving execution quality.