[Paper Review] Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

Frank Xing|arXiv (Cornell University)|Jan 11, 2024

Topic Modeling44 references4 citations

TL;DR

This paper proposes a zero-shot, training-free framework called Heterogeneous multi-Agent Discussion (HAD) that uses specialized large language model (LLM) agents to improve financial sentiment analysis (FSA) without fine-tuning. By assigning each agent a role focused on specific error types—such as mood, rhetoric, or reference—HAD achieves higher accuracy than naive prompting, especially when agent discussions are substantial, closing 25–35% of the performance gap between prompting and fine-tuning.

ABSTRACT

Large language models (LLMs) have drastically changed the possible ways to design intelligent systems, shifting the focuses from massive data acquisition and new modeling training to human alignment and strategical elicitation of the full potential of existing pre-trained models. This paradigm shift, however, is not fully realized in financial sentiment analysis (FSA), due to the discriminative nature of this task and a lack of prescriptive knowledge of how to leverage generative models in such a context. This study investigates the effectiveness of the new paradigm, i.e., using LLMs without fine-tuning for FSA. Rooted in Minsky's theory of mind and emotions, a design framework with heterogeneous LLM agents is proposed. The framework instantiates specialized agents using prior domain knowledge of the types of FSA errors and reasons on the aggregated agent discussions. Comprehensive evaluation on FSA datasets show that the framework yields better accuracies, especially when the discussions are substantial. This study contributes to the design foundations and paves new avenues for LLMs-based FSA. Implications on business and management are also discussed.

Motivation & Objective

• To address the lack of prescriptive design knowledge for leveraging generative LLMs in financial sentiment analysis (FSA) without fine-tuning.
• To improve FSA accuracy by simulating human-like reasoning through specialized LLM agents focused on common LLM error types.
• To develop a theory-informed, training-free framework that outperforms naive prompting and approaches fine-tuned models.
• To explore the effectiveness of heterogeneous agent collaboration in FSA, particularly in handling ambiguity, jargon, and external references.
• To contribute a design theory for LLM-based FSA that is generalizable across datasets and applicable in real-world financial decision-making contexts.

Proposed method

• The framework employs five heterogeneous LLM agents, each prompted to focus on a distinct error type: mood, rhetoric, aspect, reference, and dependency.
• Each agent analyzes the input text independently, generating sentiment predictions based on its specialized role.
• A shared discussion mechanism aggregates outputs from all agents, with consensus determined by majority voting.
• The design is grounded in Minsky’s theory of mind and emotions, modeling cognitive resources as specialized agents.
• The framework is zero-shot and does not require fine-tuning, relying solely on prompt engineering and agent collaboration.
• Evaluation uses multiple FSA datasets (e.g., StockSen, CMC, SEntFiN, FPB, FiQA), comparing HAD against naive prompting and fine-tuned baselines.

Experimental results

Research questions

• RQ1 Can a heterogeneous multi-agent LLM framework improve FSA accuracy without fine-tuning?
• RQ2 Does agent specialization based on known LLM error types lead to better performance than generic prompting?
• RQ3 How does the depth of agent discussion influence the final sentiment prediction accuracy?
• RQ4 Which agent roles (e.g., mood, rhetoric, reference) contribute most significantly to performance gains?
• RQ5 Can the framework close a significant portion of the performance gap between zero-shot prompting and fine-tuned models?

Key findings

• HAD improves FSA accuracy across multiple datasets, especially when agent discussions are substantial, outperforming naive prompting.
• The framework closes 25–35% of the performance gap between zero-shot prompting and fine-tuned models.
• The mood, rhetoric, and aspect agents contribute most significantly to performance gains, while the dependency agent reduces performance and can be removed for optimization.
• The framework correctly identifies neutral sentiment in ambiguous cases where naive prompting incorrectly labels messages as positive due to subtle positive phrasing.
• Case studies show HAD successfully resolves challenges such as jargon, external references, and irrealis mood, correcting errors made by single LLMs.
• The design is effective across diverse datasets, including newer ones like StockSen and SEntFiN, indicating generalizability and zero-shot transferability.