[Paper Review] A Survey of Large Language Models in Finance (FinLLMs)

Jean Lee, Nicholas Stevens+2 more|arXiv (Cornell University)|Feb 4, 2024

Stock Market Forecasting Methods14 citations

TL;DR

This survey traces the evolution from general-domain LLMs to financial-domain FinPLMs and FinLLMs, compares techniques, summarizes six benchmarks and eight advanced tasks, and discusses opportunities and challenges in FinLLMs.

ABSTRACT

Large Language Models (LLMs) have shown remarkable capabilities across a wide variety of Natural Language Processing (NLP) tasks and have attracted attention from multiple domains, including financial services. Despite the extensive research into general-domain LLMs, and their immense potential in finance, Financial LLM (FinLLM) research remains limited. This survey provides a comprehensive overview of FinLLMs, including their history, techniques, performance, and opportunities and challenges. Firstly, we present a chronological overview of general-domain Pre-trained Language Models (PLMs) through to current FinLLMs, including the GPT-series, selected open-source LLMs, and financial LMs. Secondly, we compare five techniques used across financial PLMs and FinLLMs, including training methods, training data, and fine-tuning methods. Thirdly, we summarize the performance evaluations of six benchmark tasks and datasets. In addition, we provide eight advanced financial NLP tasks and datasets for developing more sophisticated FinLLMs. Finally, we discuss the opportunities and the challenges facing FinLLMs, such as hallucination, privacy, and efficiency. To support AI research in finance, we compile a collection of accessible datasets and evaluation benchmarks on GitHub.

Motivation & Objective

• Chart the historical progression from general-domain PLMs to financial-domain FinPLMs and FinLLMs.
• Compare training and fine-tuning techniques used in FinPLMs and FinLLMs.
• Summarize benchmark performance across multiple financial NLP tasks and datasets.
• Introduce advanced financial NLP tasks and datasets to guide future FinLLM development.
• Discuss opportunities, challenges, and practical considerations for FinLLMs in real-world finance applications.

Proposed method

• Survey the evolution from GPT-series and open-source LLMs to FinLLMs and financial-domain models.
• Compare five techniques across four FinPLMs and four FinLLMs, focusing on training data, methods, and instruction fine-tuning.
• Summarize performance on six benchmark tasks and datasets and outline eight advanced financial NLP tasks and datasets.
• Compile accessible datasets and benchmarks on GitHub to support future FinLLM research.
• Discuss practical considerations such as privacy, efficiency, and hallucinations in FinLLMs.

Figure 1 : Timeline showing the evolution of selected PLM/LLM releases from the general domain to the financial domain.

Experimental results

Research questions

• RQ1 What is the historical progression from general-domain LMs to FinLLMs and what models define this trajectory?
• RQ2 What training, data, and fine-tuning techniques characterize FinPLMs and FinLLMs?
• RQ3 How do FinPLMs and FinLLMs perform on established financial NLP benchmarks and what are the gaps revealed by advanced tasks?
• RQ4 What datasets and benchmarks exist or are needed to advance FinLLMs, and how can they be leveraged for future research?

Key findings

• Mixed-domain FinPLMs show strong performance on sentiment analysis, text classification, and NER tasks.
• Task-specific SOTA models outperform FinLLMs on QA, SMP, and summarization tasks, indicating room for improvement in FinLLMs for these areas.
• GPT-4 exhibits strong performance across most benchmarks except summarization, where task-specific models prevail.
• FinMA, InvestLM, FinGPT, and BloombergGPT illustrate a spectrum of licensing, data sources, and architectural choices (LLaMA-based, BLOOM-style, etc.).
• RAG and other retrieval-based approaches are highlighted as promising directions to improve reliability and privacy in FinLLMs.

Figure 2 : Comparison of techniques used in financial LMs: from FinPLMs to FinLLMs.