[论文解读] EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

Molei Qin, Shuo Sun+4 more|arXiv (Cornell University)|Sep 22, 2023

Financial Markets and Investment Strategies被引用 4

一句话总结

EarnHFT 提出了一种用于加密货币市场高频交易的三阶段分层强化学习框架，采用 Q-teacher 提升训练效率，结合多样化趋势特异性强化学习智能体池，并通过分钟级路由机制动态选择最优智能体。在高保真度仿真中，该方法在六项金融指标上均实现比最先进基线高出 30% 的盈利能力。

ABSTRACT

High-frequency trading (HFT) uses computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performance. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second-level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.

研究动机与目标

• 解决由于轨迹极长（例如每月 240 万步）导致的高频交易（HFT）训练数据效率低下的挑战。
• 克服因市场趋势变化以及训练与测试阶段数据分布差异导致的高频交易性能下降问题。
• 通过分层分解与最优价值监督，提升在波动性加密货币市场中强化学习智能体的训练效率与鲁棒性。
• 构建一个稳定且自适应的交易系统，使其在多种市场状态（牛市、熊市、盘整、回调）下均保持高盈利能力。

提出的方法

• 在第一阶段，通过使用未来价格信息的动态规划计算 Q-teacher，作为正则化项，以提升第二层强化学习智能体的训练效率与性能。
• 在第二阶段，训练数百个具有不同收益率偏好（如牛市、熊市、盘整）的第二层强化学习智能体，并仅选择最盈利的智能体组成多样化策略池。
• 在第三阶段，使用强化学习训练一个分钟级路由智能体，根据当前市场状况从池中动态选择最合适的第二层智能体。
• 使用 DTW（动态时间规整）对市场趋势进行分类，并据此标注数据集，用于智能体训练与选择。
• 利用最优动作价值监督（OS）与最优演员（OA）加速收敛并提升最终性能，消融实验验证了其影响。
• 采用高保真度仿真环境，在多种加密资产（如 BTC、ETH、GALA）上评估系统在不同市场状态下的性能。

Figure 1: The overview of EarnHFT. First, we compute a Q-teacher for enhancing the performance and training efficiency of second-level RL agents. Then, we efficiently train diverse RL agents under various market trends and select a small fraction of them to form an agent pool based on profitability.

实验结果

研究问题

• RQ1 分层强化学习框架是否能提升在轨迹极长的高频交易中的训练效率与性能？
• RQ2 多样化趋势特异性强化学习智能体池在波动性加密货币市场中，如何提升对市场状态切换的鲁棒性？
• RQ3 最优价值监督（Q-teacher）在多大程度上能加速高频交易强化学习智能体的收敛并提升最终盈利能力？
• RQ4 基于每分钟动态选择智能体的路由机制，是否能实现优于静态或单智能体方法的性能？
• RQ5 Q-teacher、智能体池与路由机制的组合相比最先进基线，在盈利能力与风险调整回报方面表现如何？

主要发现

• EarnHFT 在六项金融指标上均优于六种最先进基线方法，在所有测试数据集中，盈利能力至少比第二名高出 30%。
• Q-teacher（OS）将 GALA 数据集上的收敛训练步数减少高达 85%，显著提升训练效率。
• 最优演员（OA）进一步提升最终收益，尽管收敛时间略有增加，但证明其在性能增强方面的价值。
• 路由机制通过根据当前市场趋势动态选择智能体实现优异性能，其选择分布反映出市场波动性——在 ETH 和 GALA 等高波动市场中更为均衡。
• 基于策略的方法（如 PPO、DRA）在熊市中因收敛至虚拟策略而失效，而基于规则的方法对止损与止盈参数高度敏感。
• 基于值的方法（如 CDQRP、DQN）仅在稳定趋势下表现良好，而 EarnHFT 在所有市场状态下（包括高波动与趋势转变条件）均保持强劲表现。

Figure 2: Trading process of EarnHFT in ETH