[Paper Review] Aligning LLMs with Human Instructions and Stock Market Feedback in Financial Sentiment Analysis

Zhao Zhang, Roy E. Welsch|arXiv (Cornell University)|Oct 19, 2024

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

This paper proposes an adaptive retrieval-augmented LLM framework that dynamically adjusts knowledge source weights in response to stock market feedback, enhancing financial sentiment analysis. By fine-tuning LLaMA 2 models with instruction tuning, RAG, and direct/RL-based weight refinement, the method achieves 1–6% higher accuracy and F1 scores and generates sentiment-driven portfolios with a 3.61% higher Sharpe ratio than S&P 500 in bull markets, while reducing drawdowns by 5x in bear markets.

ABSTRACT

Financial sentiment analysis is crucial for trading and investment decision-making. This study introduces an adaptive retrieval augmented framework for Large Language Models (LLMs) that aligns with human instructions through Instruction Tuning and incorporates market feedback to dynamically adjust weights across various knowledge sources within the Retrieval-Augmented Generation (RAG) module. Building upon foundational models like LLaMA 2, we fine-tune a series of LLMs ranging from 7B to 70B in size, enriched with Instruction Tuning and RAG, and further optimized through direct feedback and Reinforcement Learning (RL)-based refinement methods applied to the source weights of RAG.Through extensive evaluation, we demonstrate that the sentiment outputs from our LLMs more accurately mirror the intrinsic sentiment of textual data, showcasing a 1% to 6% boost in accuracy and F1 score over existing state-of-the-art models and leading conversational AI systems. Moreover, the sentiments extracted are more indicative of the directions in stock price movements. On top of that, we successfully construct portfolios that yield a 3.61% higher Sharpe ratio compared to the S&P 500 baseline in bullish markets. These portfolios also demonstrate resilience in bearish markets, with a 5x reduction in return losses compared to those typically experienced by the S&P 500.

Motivation & Objective

• To improve financial sentiment analysis by aligning LLMs with real-world market feedback to enhance predictive accuracy of sentiment outputs.
• To address the limitations of static RAG weighting by introducing dynamic, feedback-driven adaptation of knowledge source contributions.
• To investigate the impact of model scale (7B to 70B parameters) on sentiment analysis performance in financial contexts.
• To evaluate whether sentiment predictions aligned with market returns can generate superior trading signals and portfolio performance.
• To reduce reliance on keyword-based retrieval by exploring semantic search integration in future work.

Proposed method

• The framework employs Retrieval-Augmented Generation (RAG) with multiple external knowledge sources to enrich LLM context during inference.
• Knowledge source weights in RAG are dynamically adjusted using direct feedback or reinforcement learning (RL) based on subsequent stock price movements.
• Large language models (LLaMA 2 variants from 7B to 70B parameters) are fine-tuned via instruction tuning and RAG to improve financial sentiment understanding.
• Market feedback is used as a reward signal to refine RAG weights, creating a long feedback loop that improves sentiment prediction over time.
• The system evaluates sentiment outputs against actual market returns to iteratively optimize the weighting of information sources.
• A semantic search capability is proposed for future integration to replace keyword-based retrieval and improve retrieval relevance.

Experimental results

Research questions

• RQ1 How can we enhance the LLMs’ RAG module with adaptive and non-uniform weighting of multiple knowledge sources and update the weights based on real-world market feedback?
• RQ2 What is the impact of increasing model size on the performance of LLMs in financial sentiment analysis?
• RQ3 Does aligning LLMs with market feedback result in sentiment predictions that more accurately reflect near-future stock price movements?
• RQ4 How do direct feedback and RL-based weight refinement compare in optimizing RAG performance for financial sentiment tasks?
• RQ5 Can sentiment outputs from feedback-aligned LLMs generate superior trading strategies with measurable risk-adjusted returns?

Key findings

• The LLaMA I-RAG-RL model achieved a Sharpe ratio of 2.3557, representing a 3.61% improvement over the S&P 500 baseline (2.2736) in bull markets.
• All LLM portfolios outperformed the S&P 500 baseline in cumulative returns during bull markets, with the most advanced model showing the strongest risk-adjusted performance.
• In bear markets (2022), LLM portfolios reduced maximum drawdowns by 5x compared to the S&P 500, demonstrating resilience during market downturns.
• The sentiment outputs from the refined models showed a 1% to 6% improvement in accuracy and F1 score over state-of-the-art models and leading conversational AI systems.
• The direct feedback method for RAG weight refinement proved highly effective, with RL-based refinement further enhancing performance, particularly in volatile conditions.
• Sentiment predictions from the optimized LLMs were more predictive of next-day stock price movements, confirming the value of market-aligned refinement.