[Paper Review] Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks

Xianzhi Li, Chan, Samuel+5 more|arXiv (Cornell University)|May 10, 2023

Stock Market Forecasting Methods15 citations

TL;DR

The paper empirically evaluates ChatGPT and GPT-4 on eight financial NLP benchmarks across five task categories, comparing them to domain-specific models and fine-tuned baselines to assess their strengths and limitations in finance.

ABSTRACT

The most recent large language models(LLMs) such as ChatGPT and GPT-4 have shown exceptional capabilities of generalist models, achieving state-of-the-art performance on a wide range of NLP tasks with little or no adaptation. How effective are such models in the financial domain? Understanding this basic question would have a significant impact on many downstream financial analytical tasks. In this paper, we conduct an empirical study and provide experimental evidences of their performance on a wide variety of financial text analytical problems, using eight benchmark datasets from five categories of tasks. We report both the strengths and limitations of the current models by comparing them to the state-of-the-art fine-tuned approaches and the recently released domain-specific pretrained models. We hope our study can help understand the capability of the existing models in the financial domain and facilitate further improvements.

Motivation & Objective

• Assess the effectiveness of generalist LLMs (ChatGPT and GPT-4) on financial text analytics tasks.
• Compare their performance to domain-specific pretrained models and fine-tuned baselines.
• Identify strengths, limitations, and prompting strategies that affect financial NLP tasks.
• Provide actionable guidance on when to use LLMs versus fine-tuned domain models in finance.

Proposed method

• Use gpt-3.5-turbo and GPT-4 (8k context, with GPT-4 16k for some FinQA experiments) in zero-shot, few-shot, and chain-of-thought prompting.
• Evaluate on eight datasets across five task categories: sentiment analysis, classification, NER, relation extraction, and QA.
• Compare against FinBert, FinQANet, and BloombergGPT, plus baselines like CRF for NER and Luke-base for RE.
• Apply standard evaluation metrics: accuracy, macro-F1, macro-F1 (NER), and entity-level F1 where applicable.
• In QA tasks, analyze few-shot and CoT prompting impacts and compare to specialized FinQANet variants.

Experimental results

Research questions

• RQ1 Can ChatGPT and GPT-4 outperform domain-specific finetuned models on financial NLP benchmarks?
• RQ2 How do prompting strategies (zero-shot, few-shot, chain-of-thought) affect performance in finance tasks?
• RQ3 Which tasks in finance (sentiment, classification, NER, RE, QA) are well-suited for generalist LLMs, and where do domain-specific models still dominate?
• RQ4 What are the limitations of generalist LLMs in structured prediction and numerical reasoning within finance?

Key findings

• GPT-4 generally outperforms ChatGPT and many baselines across most tasks and datasets.
• Few-shot and especially Chain-of-Thought prompting significantly boost performance, sometimes by 10–30 percentage points in QA tasks.
• For NER and some structured-prediction tasks, domain-tuned models (e.g., BloombergGPT, FinQANet, Luke-base) can still outperform generalist LLMs.
• In QA tasks, GPT-4 often surpasses other models and can exceed some fine-tuned baselines, but professional-level accuracy (~90%) remains unattained.
• Generalist LLMs can outperform domain-specific models on several tasks, but their advantages are task-dependent and not uniform across all financial NLP challenges.
• Prompting strategies (few-shot, CoT) are recommended as first-line methods when applying LLMs to financial NLP.