[论文解读] FinRL-Podracer: High Performance and Scalable Deep Reinforcement Learning for Quantitative Finance

Zechu Li, Xiaoyang Liu+4 more|RePEc: Research Papers in Economics|Nov 7, 2021

Stock Market Forecasting Methods被引用 6

一句话总结

FinRL-Podracer 是一个高性能、可扩展的深度强化学习（DRL）框架，专为量化金融设计，通过在GPU云上实现RLOps（持续训练、集成与交付），显著加速了DRL驱动的交易策略开发。该框架采用代际演化机制结合集成学习与多级GPU映射，相较于RLlib、Stable Baseline 3和FinRL，在NASDAQ-100分钟级数据上实现了3×–7×的训练加速和12%–35%更高的年化收益。

ABSTRACT

Machine learning techniques are playing more and more important roles in finance market investment. However, finance quantitative modeling with conventional supervised learning approaches has a number of limitations. The development of deep reinforcement learning techniques is partially addressing these issues. Unfortunately, the steep learning curve and the difficulty in quick modeling and agile development are impeding finance researchers from using deep reinforcement learning in quantitative trading. In this paper, we propose an RLOps in finance paradigm and present a FinRL-Podracer framework to accelerate the development pipeline of deep reinforcement learning (DRL)-driven trading strategy and to improve both trading performance and training efficiency. FinRL-Podracer is a cloud solution that features high performance and high scalability and promises continuous training, continuous integration, and continuous delivery of DRL-driven trading strategies, facilitating a rapid transformation from algorithmic innovations into a profitable trading strategy. First, we propose a generational evolution mechanism with an ensemble strategy to improve the trading performance of a DRL agent, and schedule the training of a DRL algorithm onto a GPU cloud via multi-level mapping. Then, we carry out the training of DRL components with high-performance optimizations on GPUs. Finally, we evaluate the FinRL-Podracer framework for a stock trend prediction task on an NVIDIA DGX SuperPOD cloud. FinRL-Podracer outperforms three popular DRL libraries Ray RLlib, Stable Baseline 3 and FinRL, i.e., 12% \sim 35% improvements in annual return, 0.1 \sim 0.6 improvements in Sharpe ratio and 3 times \sim 7 times speed-up in training time. We show the high scalability by training a trading agent in 10 minutes with $80$ A100 GPUs, on NASDAQ-100 constituent stocks with minute-level data over 10 years.

研究动机与目标

• 解决量化金融中DRL应用面临的陡峭学习曲线与低效训练问题。
• 在可扩展的云环境中，为DRL驱动的交易策略实现持续训练、集成与交付（RLOps）。
• 通过代际演化与GPU优化的分布式训练，提升交易表现与训练速度。
• 在GPU云基础设施上，通过大规模金融数据验证框架的高可扩展性与高性能。

提出的方法

• 提出一种金融领域的RLOps范式，将建模、信号生成、投资组合优化与执行整合为单一DRL智能体。
• 采用代际演化机制与集成策略，通过跨代选择表现最优的模型，迭代提升智能体性能。
• 利用多级GPU映射技术，将DRL训练调度至GPU云环境，实现高效的并行化与硬件感知优化。
• 应用低级别GPU优化技术，包括并行化封装、GPU加速与存储优化，以实现高性能训练。
• 实施基于锦标赛的集成训练方法，结合异步并行机制，降低通信开销并提升可扩展性。
• 在NVIDIA DGX SuperPOD云平台上部署框架，对10年NASDAQ-100分钟级数据进行大规模训练。

Figure 1. Software stack for an algorithmic trading process: conventional approach vs. RLOps in finance.

实验结果

研究问题

• RQ1 一个可扩展的高性能DRL框架能否在显著缩短训练时间的同时，提升量化金融中的交易表现？
• RQ2 与单智能体训练相比，结合集成学习的代际演化机制在多大程度上提升了DRL智能体的性能？
• RQ3 GPU云基础设施在多大程度上能够加速高频金融数据的DRL训练？
• RQ4 与RLlib、Stable Baseline 3和FinRL等成熟DRL库相比，FinRL-Podracer在性能与效率方面表现如何？
• RQ5 大规模并行仿真与分布式训练能否在DRL驱动的交易中解锁更高水平的探索能力与市场动态理解？

主要发现

• 在NASDAQ-100分钟级数据上，FinRL-Podracer相较于RLlib、Stable Baseline 3和FinRL，实现了12%–35%更高的年化收益。
• 与基线DRL库相比，该框架将夏普比率提升了0.1–0.6个单位。
• 相较于RLlib，训练时间缩短了3×至7×；FinRL-Podracer在使用80张A100 GPU的情况下，仅用800秒即达到4.0的累计收益。
• 在80张A100 GPU下，FinRL-Podracer仅用10分钟即完成对10年NASDAQ-100分钟级数据的交易智能体训练，展现出极高的可扩展性。
• 代际演化机制显著优于单次训练，其演化曲线（粗绿色线）在第一代即显著超越10个独立智能体的性能。
• 该框架表现出强大的可扩展性，随着GPU数量增加，性能持续提升；而RLlib在增加GPU数量时几乎未见速度提升。

Figure 2. Overview of FinRL-Podracer that has three layers: trading task layer, evolution layer and training layer.