[Paper Review] TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution

Karush Suri, Xiao Qi Shi+2 more|arXiv (Cornell University)|Feb 16, 2021

Adversarial Robustness in Machine Learning4 citations

TL;DR

TradeR proposes a hierarchical deep reinforcement learning framework for high-frequency trade execution that addresses catastrophe and surprise minimization during abrupt market events, such as the 2019 COVID-19 stock market crash. By integrating an energy-based intrinsic motivation scheme with a surprise value function, TradeR achieves robust, profitable trading across 35 S&P500 stocks with reduced risk and stable performance.

ABSTRACT

Advances in Reinforcement Learning (RL) span a wide variety of applications which motivate development in this area. While application tasks serve as suitable benchmarks for real world problems, RL is seldomly used in practical scenarios consisting of abrupt dynamics. This allows one to rethink the problem setup in light of practical challenges. We present Trade Execution using Reinforcement Learning (TradeR) which aims to address two such practical challenges of catastrophy and surprise minimization by formulating trading as a real-world hierarchical RL problem. Through this lens, TradeR makes use of hierarchical RL to execute trade bids on high frequency real market experiences comprising of abrupt price variations during the 2019 fiscal year COVID19 stock market crash. The framework utilizes an energy-based scheme in conjunction with surprise value function for estimating and minimizing surprise. In a large-scale study of 35 stock symbols from the S&P500 index, TradeR demonstrates robustness to abrupt price changes and catastrophic losses while maintaining profitable outcomes. We hope that our work serves as a motivating example for application of RL to practical problems.

Motivation & Objective

• To address practical challenges in reinforcement learning, particularly catastrophe and surprise minimization, in real-world trading scenarios.
• To develop a hierarchical RL framework that enables stable and profitable trade execution under abrupt market dynamics.
• To incorporate intrinsic motivation via an energy-based scheme and surprise value function to guide risk-averse decision-making.
• To evaluate the framework on real high-frequency market data during the 2019 COVID-19 market crash.
• To demonstrate the effectiveness of hold actions and intrinsic motivation in emulating human-like trading behavior.

Proposed method

• TradeR employs a two-level hierarchical policy: a low-level order policy that estimates optimal trade quantities, and a high-level bid policy that executes trades based on current market states.
• An energy-based intrinsic motivation scheme estimates surprise in real-time, enabling the agent to minimize unexpected market shocks.
• The surprise value function acts as an intrinsic reward signal, guiding the agent to avoid high-surprise states and reduce catastrophic losses.
• The framework is trained on 1-minute high-frequency data from 35 S&P500 stocks during the 2019 market crash, using deep Q-networks for function approximation.
• The agent is provided with a 'hold' action, allowing it to refrain from trading when market conditions are unfavorable, improving risk control.
• Ablation studies evaluate the impact of intrinsic motivation and the hold action on performance, risk, and consistency across random seeds.

Experimental results

Research questions

• RQ1 Can a hierarchical RL framework effectively minimize surprise and catastrophic losses during abrupt market events?
• RQ2 How does incorporating an energy-based intrinsic motivation scheme improve robustness and profitability in high-frequency trading?
• RQ3 To what extent does the inclusion of a 'hold' action enhance stability and risk-averse behavior in RL-based trading agents?
• RQ4 How does the TradeR framework perform under varying initial balance levels and market volatility?
• RQ5 Does the intrinsic motivation scheme lead to more human-like trading patterns, such as buying more and selling less?

Key findings

• TradeR demonstrated consistent profitability and robustness across 35 S&P500 stocks during the 2019 market crash, even under abrupt price variations.
• The inclusion of the hold action significantly improved performance stability, allowing the agent to avoid unnecessary trades during unfavorable conditions.
• The energy-based intrinsic motivation scheme reduced the total number of shares sold and lowered variance across random seeds, indicating more risk-averse behavior.
• Performance improved with higher starting balance amounts, suggesting scalability and adaptability to increasing trading burdens.
• Ablation studies confirmed that both the surprise value function and hold action are critical for minimizing catastrophic losses and maintaining stable returns.
• The framework achieved stable learning across random seeds, indicating sample efficiency and reliability in high-stakes, high-frequency environments.