[Paper Review] Risk-Sensitive Compact Decision Trees for Autonomous Execution in Presence of Simulated Market Response

Svitlana Vyetrenko, Shaojie Xu|arXiv (Cornell University)|Jun 5, 2019

Stock Market Forecasting Methods35 references4 citations

TL;DR

This paper proposes a risk-sensitive reinforcement learning framework using compact decision trees to optimize high-volume order execution in limit order book markets. By training an agent in a simulator that models market impact and response from historical data, the method reduces execution cost by 32% while increasing cost variance by 27%, demonstrating improved risk-adjusted performance over benchmark strategies.

ABSTRACT

We demonstrate an application of risk-sensitive reinforcement learning to optimizing execution in limit order book markets. We represent taking order execution decisions based on limit order book knowledge by a Markov Decision Process; and train a trading agent in a market simulator, which emulates multi-agent interaction by synthesizing market response to our agent's execution decisions from historical data. Due to market impact, executing high volume orders can incur significant cost. We learn trading signals from market microstructure in presence of simulated market response and derive explainable decision-tree-based execution policies using risk-sensitive Q-learning to minimize execution cost subject to constraints on cost variance.

Motivation & Objective

• To develop an autonomous trading agent that minimizes risk-adjusted execution cost in limit order book markets.
• To model market impact and response from other participants using a data-driven simulator based on historical order flow.
• To generate interpretable, compact decision tree policies from risk-sensitive Q-learning for real-world deployment.
• To balance cost reduction and cost variance through risk-sensitive reinforcement learning.
• To compare the performance of decision tree-based agents against traditional non-RL benchmarks like TWAP and VWAP.

Proposed method

• The problem is modeled as a Markov Decision Process (MDP) with state variables derived from limit order book features and agent position.
• A market simulator synthesizes multi-agent market response by learning from historical data, emulating how other participants react to aggressive trades.
• Risk-sensitive Q-learning is applied with a cost function that penalizes both expected execution cost and its variance, controlled by a risk-sensitivity parameter β.
• The resulting Q-value table is converted into a decision tree using Hunt’s algorithm for interpretability and compact representation.
• State variables are discretized into bins to enable tabular Q-learning, with bin boundaries chosen to balance sample distribution across states.
• The framework is trained and validated in simulation, with performance compared to non-RL benchmark strategies like TWAP.

Experimental results

Research questions

• RQ1 Can risk-sensitive reinforcement learning improve execution cost and risk-adjusted performance in limit order book trading?
• RQ2 How does a decision tree-based policy derived from Q-learning compare to traditional execution strategies like TWAP and VWAP?
• RQ3 To what extent can a simulated market response accurately reflect real multi-agent interactions in order execution?
• RQ4 Can compact, interpretable decision trees effectively represent complex execution policies learned via RL?
• RQ5 How does the trade-off between cost reduction and cost variance manifest under risk-sensitive learning?

Key findings

• The RL-based decision tree agent achieved a 32% reduction in execution cost compared to the non-RL benchmark strategy.
• This cost reduction came with a 27% increase in the standard deviation of execution cost, reflecting a deliberate risk trade-off.
• The simulator successfully emulated market response to aggressive trades using historical data, enabling realistic multi-agent interaction modeling.
• The decision tree policy was both compact and interpretable, making it suitable for practical deployment in algorithmic trading systems.
• The risk-sensitive Q-learning approach effectively balanced cost minimization and variance control through the β parameter.
• The results demonstrate that model-free RL with interpretable policy representations can outperform static execution strategies in dynamic, impact-aware environments.