[Paper Review] Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams

Ethan Callanan, Amarachi B. Mbakwe+7 more|arXiv (Cornell University)|Oct 12, 2023

Artificial Intelligence in Healthcare and Education17 citations

TL;DR

This study evaluates ChatGPT and GPT-4 on CFA Level I/II mock exams under zero-shot, chain-of-thought, and few-shot prompting, estimates pass chances, analyzes limitations, and suggests strategies to improve financial reasoning in LLMs.

ABSTRACT

Large Language Models (LLMs) have demonstrated remarkable performance on a wide range of Natural Language Processing (NLP) tasks, often matching or even beating state-of-the-art task-specific models. This study aims at assessing the financial reasoning capabilities of LLMs. We leverage mock exam questions of the Chartered Financial Analyst (CFA) Program to conduct a comprehensive evaluation of ChatGPT and GPT-4 in financial analysis, considering Zero-Shot (ZS), Chain-of-Thought (CoT), and Few-Shot (FS) scenarios. We present an in-depth analysis of the models' performance and limitations, and estimate whether they would have a chance at passing the CFA exams. Finally, we outline insights into potential strategies and improvements to enhance the applicability of LLMs in finance. In this perspective, we hope this work paves the way for future studies to continue enhancing LLMs for financial reasoning through rigorous evaluation.

Motivation & Objective

• Assess the financial reasoning capabilities of large language models on CFA mock exam questions.
• Compare ChatGPT and GPT-4 under Zero-Shot, Chain-of-Thought, and Few-Shot prompting settings.
• Estimate each model's likelihood of passing CFA Level I and Level II under different prompts.
• Analyze error modes and topic-level strengths/weaknesses to inform improvements in financial reasoning.
• Propose strategies to enhance LLMs for finance, including tool integration and retrieval-augmented approaches.

Proposed method

• Use CFA Level I (5 mock exams) and Level II (2 mock exams) as the evaluation dataset.
• Test prompting paradigms: Zero-Shot, Chain-of-Thought, and Few-Shot (with various shot selection strategies).
• Apply OpenAI ChatCompletion API (gpt-3.5-turbo and GPT-4) with temperature set to zero to reduce randomness.
• Conduct memorization checks to ensure questions are not in training data.
• Measure accuracy against official solution sets as the sole evaluation metric.
• Discuss topic- and level-specific performance, error modes, and potential improvements.

Experimental results

Research questions

• RQ1 How do ChatGPT and GPT-4 perform on CFA Level I and Level II questions across prompting paradigms?
• RQ2 Do Chain-of-Thought or Few-Shots prompting significantly improve performance, and under what conditions?
• RQ3 Could these models plausibly pass CFA Level I and Level II under proposed criteria?
• RQ4 What are the dominant error modes and topic-specific strengths/weaknesses of the models in financial reasoning?

Key findings

• GPT-4 generally outperforms ChatGPT across topics and levels and prompts.
• Level II is harder than Level I for both models due to longer prompts and more table-based calculations.
• CoT prompting yields limited improvements; it helps GPT-4 more in Level II but can hurt ChatGPT in Level I.
• Few-Shot prompting provides notable gains, with 2S/10S showing strong performance depending on level and model.
• Under proposed pass criteria, GPT-4 has a plausible chance to pass Level I and Level II with Few-Shot and/or CoT prompting, while ChatGPT is less likely.
• Common error modes include knowledge gaps, calculation mistakes, and inconsistency; CoT can magnify knowledge gaps in some cases.