[Paper Review] FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents

Mostapha Benhenda|ArXiv.org|Feb 11, 2025

FinTech, Crowdfunding, Digital Finance9 citations

TL;DR

The paper introduces a risk-sensitive trading agent that augments CVaR-PPO with LLM-derived stock recommendations and risk signals from financial news, evaluated on Nasdaq-100 with multiple LLMs.

ABSTRACT

This paper presents a novel risk-sensitive trading agent combining reinforcement learning and large language models (LLMs). We extend the Conditional Value-at-Risk Proximal Policy Optimization (CPPO) algorithm, by adding risk assessment and trading recommendation signals generated by a LLM from financial news. Our approach is backtested on the Nasdaq-100 index benchmark, using financial news data from the FNSPID dataset and the DeepSeek V3, Qwen 2.5 and Llama 3.3 language models. The code, data, and trading agents are available at: https://github.com/benstaf/FinRL_DeepSeek

Motivation & Objective

• Integrate financial news as a risk and recommendation signal into reinforcement learning for trading.
• Extend CPPO with CVaR constraints to manage downside risk in trading trajectories.
• Demonstrate LLM-based feature extraction from news beyond sentiment analysis for trading decisions.

Proposed method

• Extend CVaR-PPO by incorporating LLM-derived stock recommendations and risk scores.
• Infuse action perturbation S_f from LLM stock scores to modulate trading signals.
• Infuse risk perturbation R_f from LLM-derived news risk scores to adjust trajectory returns (D_Rf = R_f * D).
• Use representative sampling of FNSPID news per stock per day to generate prompts for three LLMs (DeepSeek-V3, Qwen 2.5, Llama 3.3).
• Backtest on Nasdaq-100 with 2019–2023 historical window and evaluate information ratio, CVaR, and Rachev ratio across PPO/CPPO and their DeepSeek variants.

Experimental results

Research questions

• RQ1 Can LLM-derived stock recommendations and risk scores improve risk-sensitive RL trading performance beyond standard PPO/CPPO baselines?
• RQ2 What is the impact of varying LLM infusion strength on trading performance in both PPO-DeepSeek and CPPO-DeepSeek configurations?
• RQ3 Does longer training duration stabilize and improve performance when LLM signals are integrated?
• RQ4 How do LLM-based risk signals affect the CVaR-constrained learning objective in trading?

Key findings

• LLM-infused PPO/CPPO show improved cumulative returns with longer training periods but may still underperform Nasdaq-100 in some setups.
• With 2M training steps, PPO-DeepSeek and CPPO-DeepSeek variants achieve notable information ratios and CVaR figures across runs, sometimes exceeding Nasdaq-100 in bear markets.
• Stronger LLM infusion generally degrades PPO performance but improves CPPO-DeepSeek performance in certain runs, indicating model- and regime-dependent effects.
• PPO-DeepSeek tends to excel in bull markets, while CPPO-DeepSeek shows strength in bear markets across two runs.
• Infusion strength experiments reveal that 10% infusion often harms PPO-DeepSeek relative to baseline, while CPPO-DeepSeek can benefit from higher infusion in some configurations.
• Table shows: PPO (100 epochs) Information Ratio 0.0100, CVaR -0.0394, Rachev 1.0637; CPPO (100 epochs) -0.0148, -0.0439, 1.0404; PPO-DeepSeek (100 epochs) -0.0093, -0.0338, 0.9890; CPPO-DeepSeek (100 epochs) 0.0078, -0.0437, 0.9818.
• S_f and R_f perturbations influence trading actions and risk-adjusted returns, with near-1 perturbations preserving stability while aligning with news signals.