[Paper Review] Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination

Haoqiang Kang, Xiao-Yang Liu|arXiv (Cornell University)|Nov 27, 2023

Stock Market Forecasting Methods5 citations

TL;DR

This paper empirically investigates hallucination in large language models (LLMs) within finance, evaluating their accuracy in explaining financial concepts and querying historical stock prices. It finds that off-the-shelf LLMs exhibit severe hallucination, but prompt-based tool learning and RAG significantly improve factual accuracy, with tool-augmented models achieving 100% accuracy in stock price queries.

ABSTRACT

The hallucination issue is recognized as a fundamental deficiency of large language models (LLMs), especially when applied to fields such as finance, education, and law. Despite the growing concerns, there has been a lack of empirical investigation. In this paper, we provide an empirical examination of LLMs' hallucination behaviors in financial tasks. First, we empirically investigate LLM model's ability of explaining financial concepts and terminologies. Second, we assess LLM models' capacity of querying historical stock prices. Third, to alleviate the hallucination issue, we evaluate the efficacy of four practical methods, including few-shot learning, Decoding by Contrasting Layers (DoLa), the Retrieval Augmentation Generation (RAG) method and the prompt-based tool learning method for a function to generate a query command. Finally, our major finding is that off-the-shelf LLMs experience serious hallucination behaviors in financial tasks. Therefore, there is an urgent need to call for research efforts in mitigating LLMs' hallucination.

Motivation & Objective

• To empirically examine hallucination in LLMs when applied to financial tasks, particularly in explaining financial terminology and querying historical stock prices.
• To assess the reliability of off-the-shelf LLMs in handling time-sensitive, factually precise financial queries.
• To evaluate the effectiveness of four mitigation strategies—few-shot prompting, DoLa decoding, RAG, and prompt-based tool learning—in reducing hallucination.
• To demonstrate that hallucination remains a critical barrier to deploying LLMs in real-world financial applications without external fact-checking mechanisms.
• To advocate for targeted research in hallucination mitigation to ensure factual reliability in high-stakes financial AI systems.

Proposed method

• Conducted empirical evaluation on LLMs' ability to explain financial concepts and terminologies using zero-shot and few-shot prompting.
• Designed a benchmark task to test LLMs' accuracy in retrieving historical stock prices using real-world queries.
• Evaluated four mitigation techniques: few-shot prompting, Decoding by Contrasting Layers (DoLa), Retrieval-Augmented Generation (RAG), and prompt-based tool learning for function call generation.
• Implemented prompt-based tool learning to enable LLMs to generate correct API queries for accessing up-to-date financial data.
• Used a zero-shot and few-shot setting to compare model performance across different LLMs, including Llama2-7B, Llama2-13B, GPT-3.5-turbo, and GPT-4.
• Measured factual accuracy through exact match evaluation between model outputs and ground-truth stock price responses.

Experimental results

Research questions

• RQ1 To what extent do off-the-shelf LLMs hallucinate when explaining financial concepts and terminologies?
• RQ2 How accurately can LLMs retrieve historical stock prices without external tools or retrieval mechanisms?
• RQ3 How effective are few-shot prompting, DoLa decoding, RAG, and prompt-based tool learning in reducing hallucination in financial LLM tasks?
• RQ4 Can prompt-based tool learning achieve near-perfect accuracy in financial query tasks, and how does it compare to other methods?
• RQ5 What are the limitations of DoLa and few-shot prompting in addressing knowledge gaps related to financial data?

Key findings

• Off-the-shelf LLMs exhibit severe hallucination in financial tasks, with significant factual errors in explaining financial terms and retrieving stock prices.
• Llama2-7B and Llama2-7B-chat models achieve near-zero accuracy in zero-shot stock price queries without external tools, highlighting their unreliability for factual tasks.
• Prompt-based tool learning enables Llama2-7B and Llama2-7B-chat to achieve 100.00% accuracy in stock price queries using only one-shot example, demonstrating its high efficacy.
• GPT-3.5-turbo and GPT-4 fail to answer stock price queries in zero-shot mode but achieve perfect accuracy when augmented with prompt-based tool learning.
• Few-shot prompting improves question-answering format adherence but does not significantly enhance factual accuracy, indicating limited impact on hallucination reduction.
• DoLa decoding shows limited improvement in factual accuracy, especially when models lack sufficient knowledge in their pretraining data, such as specific stock ticker information.