[Paper Review] Style Miner: Find Significant and Stable Explanatory Factors in Time Series with Constrained Reinforcement Learning

Dapeng Li, Feiyang Pan+4 more|arXiv (Cornell University)|Mar 21, 2023

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

Style Miner is a constrained reinforcement learning framework that discovers significant and stable style factors in high-dimensional time series by formulating the problem as a Constrained Markov Decision Process (CMDP), where explanatory power is maximized as the return and temporal stability is enforced via a constraint on autocorrelation. It achieves a 10% relative improvement in R-squared explanatory power over expert-designed factors on real financial data.

ABSTRACT

In high-dimensional time-series analysis, it is essential to have a set of key factors (namely, the style factors) that explain the change of the observed variable. For example, volatility modeling in finance relies on a set of risk factors, and climate change studies in climatology rely on a set of causal factors. The ideal low-dimensional style factors should balance significance (with high explanatory power) and stability (consistent, no significant fluctuations). However, previous supervised and unsupervised feature extraction methods can hardly address the tradeoff. In this paper, we propose Style Miner, a reinforcement learning method to generate style factors. We first formulate the problem as a Constrained Markov Decision Process with explanatory power as the return and stability as the constraint. Then, we design fine-grained immediate rewards and costs and use a Lagrangian heuristic to balance them adaptively. Experiments on real-world financial data sets show that Style Miner outperforms existing learning-based methods by a large margin and achieves a relatively 10% gain in R-squared explanatory power compared to the industry-renowned factors proposed by human experts.

Motivation & Objective

• To address the challenge of identifying low-dimensional, stable, and highly explanatory style factors in high-dimensional time-series data.
• To balance the trade-off between explanatory power and temporal stability, which existing supervised and unsupervised methods fail to optimize jointly.
• To develop a reinforcement learning-based method that can discover meaningful style factors without relying on differentiable loss functions or expert priors.
• To enable automated, data-driven discovery of style factors in complex domains such as finance and climatology.

Proposed method

• Formulates style factor discovery as a Constrained Markov Decision Process (CMDP), with explanatory power as the return and autocorrelation of the factor sequence as the constraint.
• Designs fine-grained immediate rewards based on individual stock contributions to improve credit assignment and alleviate sparse reward issues.
• Introduces a cost term based on negative autocorrelation to explicitly penalize unstable factor sequences.
• Employs a Lagrangian heuristic with adaptive penalty coefficient to dynamically balance reward maximization and constraint satisfaction.
• Uses a GRU-based policy network to model temporal dependencies and generate continuous style factors from raw time-series inputs.
• Applies a multi-task learning strategy with input features from both raw data and expert factors to improve generalization and stability.

Experimental results

Research questions

• RQ1 Can reinforcement learning effectively discover style factors that balance high explanatory power and temporal stability in time series?
• RQ2 How does the proposed constrained RL framework compare to supervised and unsupervised baselines in terms of R-squared and autocorrelation?
• RQ3 What is the impact of different input features (raw data, expert factors, or both) on the performance and stability of the learned style factors?
• RQ4 How do individual components of the method (e.g., GRU, constraint, reward decomposition) contribute to the overall performance?

Key findings

• Style Miner achieves a 10% relative improvement in R-squared explanatory power compared to industry-standard expert-designed factors on financial time series.
• On the S&P 500 dataset, Style Miner achieves an R-squared of 27.4% with an autocorrelation of 0.92, significantly outperforming DRM and other RL baselines.
• The ablation study confirms that each component—GRU, constraint, penalty, and reward decomposition—is essential, with reward decomposition contributing a 11% drop in R-squared when removed.
• When using both raw data and expert factors as input, Style Miner achieves an R-squared of 27.6% and autocorrelation of 0.93, demonstrating improved performance and stability.
• On the CSI500 dataset, Style Miner achieves 31.1% R-squared with 0.92 autocorrelation, outperforming PPO (28.7%) and DDPG (28.0%) in both explanatory power and stability.
• The adaptive penalty mechanism enables better balance between reward and constraint, achieving higher R-squared than non-adaptive counterparts while maintaining high autocorrelation.