[Paper Review] EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

Molei Qin, Shuo Sun+4 more|arXiv (Cornell University)|Sep 22, 2023

Financial Markets and Investment Strategies4 citations

TL;DR

EarnHFT proposes a three-stage hierarchical reinforcement learning framework for high-frequency trading in cryptocurrency markets, using a Q-teacher for training efficiency, a diverse pool of trend-specific RL agents, and a minute-level router to dynamically select optimal agents. It achieves 30% higher profitability than state-of-the-art baselines across six financial metrics in high-fidelity simulations.

ABSTRACT

High-frequency trading (HFT) uses computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performance. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second-level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.

Motivation & Objective

• Address the challenge of training data inefficiency in high-frequency trading (HFT) due to extremely long trajectories (e.g., 2.4 million steps/month).
• Overcome performance degradation in HFT caused by market trend shifts and data distribution shifts between training and testing.
• Improve training efficiency and robustness of RL agents in volatile crypto markets through hierarchical decomposition and optimal value supervision.
• Develop a stable, adaptive trading system that maintains high profitability across diverse market regimes (bull, bear, sideways, pullback).

Proposed method

• In Stage I, compute a Q-teacher via dynamic programming using future price information to serve as a regularizer, enhancing training efficiency and performance of second-level RL agents.
• In Stage II, train hundreds of second-level RL agents with varying return rate preferences (e.g., bull, bear, sideways) and select only the most profitable agents to form a diversified strategy pool.
• In Stage III, train a minute-level router using RL to dynamically select the most suitable second-level agent from the pool based on current market conditions.
• Use DTW (Dynamic Time Warping) to classify market trends and label datasets accordingly for agent training and selection.
• Leverage optimal action value supervision (OS) and optimal actor (OA) to accelerate convergence and improve final performance, with ablation studies validating their impact.
• Employ high-fidelity simulation environments to evaluate performance across multiple crypto assets (e.g., BTC, ETH, GALA) under diverse market regimes.

Figure 1: The overview of EarnHFT. First, we compute a Q-teacher for enhancing the performance and training efficiency of second-level RL agents. Then, we efficiently train diverse RL agents under various market trends and select a small fraction of them to form an agent pool based on profitability.

Experimental results

Research questions

• RQ1 Can a hierarchical RL framework improve training efficiency and performance in high-frequency trading with extremely long trajectories?
• RQ2 How can a diverse pool of trend-specific RL agents enhance robustness across shifting market regimes in volatile crypto markets?
• RQ3 To what extent does optimal value supervision (Q-teacher) accelerate convergence and improve final profitability in HFT RL agents?
• RQ4 Can a dynamic router that selects agents per minute achieve superior performance compared to static or single-agent approaches?
• RQ5 How does the combination of Q-teacher, agent pool, and router compare to state-of-the-art baselines in terms of profitability and risk-adjusted returns?

Key findings

• EarnHFT outperforms six state-of-the-art baselines across six financial criteria, achieving at least 30% higher profitability than the runner-up in all tested datasets.
• The Q-teacher (OS) reduces training steps to convergence by up to 85% on the GALA dataset, significantly improving training efficiency.
• The optimal actor (OA) further increases final return, though at the cost of slightly longer convergence time, demonstrating its value in performance enhancement.
• The router achieves superior performance by dynamically selecting agents based on current market trends, with selection distributions reflecting market volatility—more balanced in high-volatility markets like ETH and GALA.
• Policy-based methods (e.g., PPO, DRA) fail in bear markets due to convergence to dummy policies, while rule-based methods are highly sensitive to stop-loss and take-profit parameters.
• Value-based methods (e.g., CDQRP, DQN) perform well only under stable trends, but EarnHFT maintains strong performance across all market regimes, including volatile and trend-shifting conditions.

Figure 2: Trading process of EarnHFT in ETH