[Paper Review] Optimizing Trading Strategies in Quantitative Markets using Multi-Agent Reinforcement Learning

Hengxi Zhang, Zhendong Shi+4 more|arXiv (Cornell University)|Mar 15, 2023

Financial Markets and Investment StrategiesEconomics, Econometrics and Finance3 citations

TL;DR

This paper proposes CPPI-MADDPG and TIPP-MADDPG, novel multi-agent reinforcement learning frameworks that integrate constant proportion portfolio insurance (CPPI) and time-invariant portfolio protection (TIPP) strategies with the multi-agent deep deterministic policy gradient (MADDPG) algorithm. Evaluated on 100 real stocks from the Shenzhen Stock Exchange, both methods outperform traditional strategies and baseline RL models, with TIPP-MADDPG achieving the highest annual return (9.68%) and CPPI-MADDPG delivering the best Sharpe ratio (2.18).

ABSTRACT

Quantitative markets are characterized by swift dynamics and abundant uncertainties, making the pursuit of profit-driven stock trading actions inherently challenging. Within this context, reinforcement learning (RL), which operates on a reward-centric mechanism for optimal control, has surfaced as a potentially effective solution to the intricate financial decision-making conundrums presented. This paper delves into the fusion of two established financial trading strategies, namely the constant proportion portfolio insurance (CPPI) and the time-invariant portfolio protection (TIPP), with the multi-agent deep deterministic policy gradient (MADDPG) framework. As a result, we introduce two novel multi-agent RL (MARL) methods, CPPI-MADDPG and TIPP-MADDPG, tailored for probing strategic trading within quantitative markets. To validate these innovations, we implemented them on a diverse selection of 100 real-market shares. Our empirical findings reveal that the CPPI-MADDPG and TIPP-MADDPG strategies consistently outpace their traditional counterparts, affirming their efficacy in the realm of quantitative trading.

Motivation & Objective

• To address the challenges of high-frequency, uncertain, and dynamic quantitative trading environments where human traders are prone to irrational decisions.
• To improve portfolio management in volatile markets by integrating established financial risk-control strategies—CPPI and TIPP—into a multi-agent reinforcement learning framework.
• To evaluate whether the fusion of rule-based portfolio protection mechanisms with MARL can yield superior risk-adjusted returns compared to standard RL and traditional portfolio methods.
• To demonstrate the efficacy of CPPI-MADDPG and TIPP-MADDPG in real-market conditions using empirical backtesting on 100 stocks from the Shenzhen Stock Exchange.

Proposed method

• The framework models strategic trading as a stochastic game involving N agents, each observing a shared state vector s = [p, h, b] representing stock prices, holdings, and cash balance.
• Each agent uses a policy network πθi to select actions (portfolio weights) based on its observation, with actions constrained to follow either CPPI or TIPP rule-based insurance logic.
• The MADDPG algorithm is adapted to include centralized critics that observe joint actions and states, enabling credit assignment and improved joint policy learning.
• A replay buffer stores transitions ⟨s, a, r, s′⟩, and target networks are updated using a soft update rule with τ = 0.01 to stabilize training.
• The action selection process includes exploration via a noise process 𝒩, and actions are clipped or adjusted to satisfy CPPI/TIPP constraints if they fall outside the rule-based policy Φ(s).
• The critic is trained to minimize the loss in Equation (4), while the actor is updated using the policy gradient in Equation (3), both using a double-Q target estimation approach.

Experimental results

Research questions

• RQ1 Can the integration of CPPI and TIPP risk-control mechanisms with multi-agent deep reinforcement learning improve portfolio performance in quantitative trading environments?
• RQ2 How do CPPI-MADDPG and TIPP-MADDPG compare to baseline RL methods (e.g., MADDPG, MADQN) and traditional strategies (e.g., Universal Portfolios) in terms of risk-adjusted returns?
• RQ3 Does the inclusion of rule-based portfolio insurance enhance stability and reduce drawdowns compared to unconstrained RL agents?
• RQ4 How do the distinct risk profiles of CPPI (gap risk) and TIPP (time-based floor) influence the performance of the hybrid MARL models?
• RQ5 To what extent can the hybrid models adapt to different investor risk preferences through parameter tuning?

Key findings

• TIPP-MADDPG achieved the highest annual return of 9.68%, significantly outperforming the Universal Portfolio (3.36%) and MADDPG (8.22%).
• CPPI-MADDPG delivered the best Sharpe ratio of 2.18, indicating superior risk-adjusted performance compared to TIPP-MADDPG (2.09) and MADDPG (1.99).
• CPPI-MADDPG recorded the lowest maximum drawdown at 6.60%, demonstrating stronger downside protection than TIPP-MADDPG (9.02%) and MADDPG (12.26%).
• The TIPP-MADDPG strategy exhibited a sharp performance surge between the 80th and 160th trading days, suggesting responsiveness to market trends.
• MADDPG-based agents showed more uniform asset allocation, while CPPI-MADDPG and TIPP-MADDPG agents adopted sparser, rule-informed allocations, reflecting strategic risk control.
• The hybrid models successfully reverted to baseline MADDPG performance under specific parameter settings, confirming their flexibility and controllability for risk preference tuning.