[Paper Review] PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets

Shuo Sun, Molei Qin+2 more|arXiv (Cornell University)|Jan 14, 2023

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

This paper introduces PRUDEX-Compass, a systematic 6-axis evaluation framework for reinforcement learning in financial markets, encompassing Profitability, Risk-control, Universality, Diversity, Reliability, and Explainability across 17 metrics. It proposes AlphaMix+, a mixture-of-experts and risk-aware RL method that outperforms existing FinRL baselines across all axes on long-term real-world datasets, and releases open-source tools to accelerate trustworthy FinRL research and deployment.

ABSTRACT

The financial markets, which involve more than $90 trillion market capitals, attract the attention of innumerable investors around the world. Recently, reinforcement learning in financial markets (FinRL) has emerged as a promising direction to train agents for making profitable investment decisions. However, the evaluation of most FinRL methods only focuses on profit-related measures and ignores many critical axes, which are far from satisfactory for financial practitioners to deploy these methods into real-world financial markets. Therefore, we introduce PRUDEX-Compass, which has 6 axes, i.e., Profitability, Risk-control, Universality, Diversity, rEliability, and eXplainability, with a total of 17 measures for a systematic evaluation. Specifically, i) we propose AlphaMix+ as a strong FinRL baseline, which leverages mixture-of-experts (MoE) and risk-sensitive approaches to make diversified risk-aware investment decisions, ii) we evaluate 8 FinRL methods in 4 long-term real-world datasets of influential financial markets to demonstrate the usage of our PRUDEX-Compass, iii) PRUDEX-Compass together with 4 real-world datasets, standard implementation of 8 FinRL methods and a portfolio management environment is released as public resources to facilitate the design and comparison of new FinRL methods. We hope that PRUDEX-Compass can not only shed light on future FinRL research to prevent untrustworthy results from stagnating FinRL into successful industry deployment but also provide a new challenging algorithm evaluation scenario for the reinforcement learning (RL) community.

Motivation & Objective

• To address the lack of systematic evaluation in existing FinRL methods, which primarily focus on profit metrics while neglecting risk, reliability, and explainability.
• To develop a comprehensive benchmark that enables fair and trustworthy comparison of FinRL algorithms across multiple critical financial performance dimensions.
• To provide a strong, robust baseline (AlphaMix+) that emulates real-world trading strategies and outperforms market averages across all evaluation axes.
• To release open-source datasets, code, environment, and visualization tools to accelerate reproducible and industry-ready FinRL research.

Proposed method

• Design a 6-axis evaluation framework—PRUDEX-Compass—comprising Profitability, Risk-control, Universality, Diversity, Reliability, and eXplainability, with 17 distinct performance measures.
• Propose AlphaMix+, a deep reinforcement learning agent using a mixture-of-experts architecture to enable dynamic policy specialization and risk-sensitive Bellman backups for improved robustness.
• Implement a standardized portfolio management environment and integrate it with 4 long-term real-world financial datasets spanning over 15 years.
• Apply systematic evaluation across 8 FinRL methods, including AlphaMix+, on multiple trading tasks to validate the framework’s utility and robustness.
• Develop visualization toolkits and a public GitHub repository to support reproducibility and community adoption of the evaluation framework.
• Integrate auxiliary techniques such as volatility-aware learning and multi-objective RL to enhance model awareness of extreme market conditions.

Figure 1 : Illustration of our FinRL methods evaluation benchmark PRUDEX-Compass. The inner star plot provides a visual indication of the relative strength of different FinRL methods in terms of six axes. A mark on the star plot’s inner circle suggests the market average 3 3 3 Market average indicat

Experimental results

Research questions

• RQ1 How do existing FinRL methods perform across dimensions beyond profitability, such as risk-control, reliability, and explainability?
• RQ2 Can a single FinRL agent achieve superior performance across all 6 axes of PRUDEX-Compass without sacrificing robustness in volatile or changing market regimes?
• RQ3 To what extent does AlphaMix+ outperform existing FinRL baselines in terms of risk-adjusted returns, adaptability, and interpretability?
• RQ4 How effective is PRUDEX-Compass in identifying overfitting and poor generalization in FinRL models that perform well only on backtests?
• RQ5 Can the framework be generalized to support non-RL methods and broader machine learning applications in finance?

Key findings

• AlphaMix+ significantly outperforms all 7 other FinRL methods across all 6 axes of PRUDEX-Compass, achieving superior performance in profitability, risk-control, and explainability.
• The evaluation reveals that many existing FinRL methods, such as SAC and IMIT, suffer from poor risk-control and catastrophic drawdowns during black swan events, despite strong backtest returns.
• PRUDEX-Compass successfully identifies overfitting and lack of robustness in profit-seeking agents, demonstrating its value in filtering out unreliable methods before real-world deployment.
• The framework enables detection of performance degradation under market regime shifts, highlighting the importance of universality and reliability in real-world applications.
• The open-source release of datasets, code, and visualization tools has already facilitated reproducibility and benchmarking, with AlphaMix+ serving as a new strong baseline for future FinRL research.
• Ablation studies confirm that both the mixture-of-experts and risk-aware Bellman backup components in AlphaMix+ are essential for its superior performance across evaluation axes.

Figure 2 : Workflow of real-world trading firms.