[Paper Review] FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance

Xiao-Yang Liu, Hongyang Yang+5 more|arXiv (Cornell University)|Nov 19, 2020

Stock Market Forecasting Methods8 citations

TL;DR

FinRL is an open-source deep reinforcement learning library designed to streamline automated stock trading in quantitative finance, offering pre-configured environments, fine-tuned DRL algorithms (e.g., PPO, TD3), and reproducible tutorials for single stock, multiple stock, and portfolio trading. It achieves superior risk-adjusted returns—e.g., Sharpe ratios up to 1.49 in single-stock trading—by integrating market frictions like transaction costs and turbulence-based risk control.

ABSTRACT

As deep reinforcement learning (DRL) has been recognized as an effective approach in quantitative finance, getting hands-on experiences is attractive to beginners. However, to train a practical DRL trading agent that decides where to trade, at what price, and what quantity involves error-prone and arduous development and debugging. In this paper, we introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies. Along with easily-reproducible tutorials, FinRL library allows users to streamline their own developments and to compare with existing schemes easily. Within FinRL, virtual environments are configured with stock market datasets, trading agents are trained with neural networks, and extensive backtesting is analyzed via trading performance. Moreover, it incorporates important trading constraints such as transaction cost, market liquidity and the investor's degree of risk-aversion. FinRL is featured with completeness, hands-on tutorial and reproducibility that favors beginners: (i) at multiple levels of time granularity, FinRL simulates trading environments across various stock markets, including NASDAQ-100, DJIA, S&P 500, HSI, SSE 50, and CSI 300; (ii) organized in a layered architecture with modular structure, FinRL provides fine-tuned state-of-the-art DRL algorithms (DQN, DDPG, PPO, SAC, A2C, TD3, etc.), commonly-used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the reproducibility, and (iii) being highly extendable, FinRL reserves a complete set of user-import interfaces. Furthermore, we incorporated three application demonstrations, namely single stock trading, multiple stock trading, and portfolio allocation. The FinRL library will be available on Github at link https://github.com/AI4Finance-LLC/FinRL-Library.

Motivation & Objective

• To reduce the high barrier to entry for beginners in quantitative finance by providing a complete, beginner-friendly DRL framework for stock trading.
• To address the arduous and error-prone development process of training DRL agents for trading by offering modular, reusable components.
• To ensure reproducibility and transparency in DRL-based trading strategies through standardized environments, algorithms, and evaluation metrics.
• To support practical, real-world trading constraints such as transaction costs, market liquidity, and risk-aversion via turbulence indexing.
• To demonstrate the effectiveness of DRL in diverse trading scenarios—single stock, multiple stock, and portfolio allocation—through reproducible, end-to-end tutorials.

Proposed method

• FinRL implements a three-layered architecture: environment layer simulates real stock markets (e.g., S&P 500, NASDAQ-100) using historical data with features like price, volume, and technical indicators.
• The agent layer integrates state-of-the-art DRL algorithms including PPO, TD3, DDPG, SAC, A2C, and DQN, with hyperparameters tuned for financial trading tasks.
• Reward functions are standardized and designed to reflect risk-adjusted performance, such as Sharpe ratio or cumulative returns, to guide policy learning.
• Market frictions such as transaction costs and risk-aversion are modeled using the financial turbulence index, defined as $\text{turbulence}_{t} = (\mathbf{y}_t - \boldsymbol{\mu})\boldsymbol{\Sigma}^{-1}(\mathbf{y}_t - \boldsymbol{\mu})^\top$, which triggers risk mitigation actions when volatility spikes.
• The library supports automated backtesting and evaluation using standard benchmarks, enabling comparison across strategies and algorithms.
• Applications are demonstrated via Jupyter notebook tutorials that guide users through environment setup, agent training, and performance analysis for single stock, multiple stock, and portfolio allocation tasks.

Experimental results

Research questions

• RQ1 Can a unified, beginner-friendly DRL library significantly reduce the complexity and development time for building automated stock trading strategies in quantitative finance?
• RQ2 How do standard DRL algorithms like PPO and TD3 perform in real-world backtests across diverse market indices when incorporating realistic market frictions?
• RQ3 To what extent can risk-aversion mechanisms, such as turbulence-based action throttling, improve robustness during market stress events like the 2020 COVID-19 crash?
• RQ4 Can the FinRL library achieve reproducible, high-performing trading strategies across different trading scenarios—single stock, multi-asset, and portfolio allocation—using standardized components?
• RQ5 How does the integration of transaction costs and liquidity constraints affect the performance and stability of DRL-driven trading agents compared to frictionless benchmarks?

Key findings

• In single-stock trading using PPO, the FinRL agent achieved a Sharpe ratio of 1.49 for SPY (S&P 500 ETF), outperforming the market benchmark (S&P 500) with a Sharpe ratio of 0.74.
• For multiple stock trading on the DJIA 30 constituents, the TD3 agent achieved an annualized return of 21.40% with a Sharpe ratio of 1.38, exceeding both the DJIA index (10.61%) and the minimum-variance portfolio (8.38%).
• The DDPG agent in multi-asset trading achieved a final portfolio value of $1,381,120 from an initial $1M, with a 17.61% annualized return and a 1.03 Sharpe ratio.
• The FinRL library successfully reduced development overhead by providing pre-built, modular components and Jupyter notebook tutorials that enable end-to-end reproducibility of DRL trading strategies.
• Performance evaluation across multiple use cases showed that DRL agents incorporating market frictions (e.g., transaction costs, turbulence control) exhibited more robust and realistic behavior during volatile market conditions.
• The library’s modular design and standardized evaluation framework enabled direct comparison between algorithms, with TD3 and PPO consistently outperforming baseline strategies in risk-adjusted returns.