[Paper Review] Large Language Model in Financial Regulatory Interpretation

Zhiyu Cao, Zachary Feinstein|arXiv (Cornell University)|May 10, 2024

FinTech, Crowdfunding, Digital Finance4 citations

TL;DR

This paper proposes a systematic framework for using Large Language Models (LLMs) like GPT-4 to interpret complex financial regulations—specifically Basel III market risk rules—by designing precise prompts and optimizing document loading methods. It demonstrates that with engineered prompts and image-based document input, GPT-4 achieves 96.5% accuracy in identifying correlations and 85% in bucket classification, significantly outperforming naive prompting and other LLMs.

ABSTRACT

This study explores the innovative use of Large Language Models (LLMs) as analytical tools for interpreting complex financial regulations. The primary objective is to design effective prompts that guide LLMs in distilling verbose and intricate regulatory texts, such as the Basel III capital requirement regulations, into a concise mathematical framework that can be subsequently translated into actionable code. This novel approach aims to streamline the implementation of regulatory mandates within the financial reporting and risk management systems of global banking institutions. A case study was conducted to assess the performance of various LLMs, demonstrating that GPT-4 outperforms other models in processing and collecting necessary information, as well as executing mathematical calculations. The case study utilized numerical simulations with asset holdings -- including fixed income, equities, currency pairs, and commodities -- to demonstrate how LLMs can effectively implement the Basel III capital adequacy requirements. Keywords: Large Language Models, Prompt Engineering, LLMs in Finance, Basel III, Minimum Capital Requirements, LLM Ethics

Motivation & Objective

• To address the challenge of interpreting highly complex, verbose financial regulations such as Basel III using LLMs.
• To identify and mitigate key limitations of LLMs in financial regulatory interpretation, including poor mathematical reasoning and sensitivity to prompt design.
• To develop a systematic framework for prompt engineering and document loading that enhances LLM accuracy in extracting regulatory requirements.
• To evaluate and compare the performance of multiple LLMs—GPT-4, GPT-3.5, Claude-3, Gemini-1.5—on regulatory text interpretation tasks.
• To validate the framework through a case study on Basel III’s minimum capital requirements for market risk, using realistic asset holdings.

Proposed method

• Designed a structured prompt engineering methodology incorporating key regulatory elements (e.g., risk buckets, weights, correlations) to guide LLMs toward accurate interpretation.
• Evaluated document loading methods by comparing PDF and image inputs for regulatory documents, leveraging visual parsing advantages of image formats.
• Conducted comparative LLM performance testing across GPT-3.5, GPT-4, Claude-3-Opus, and Gemini-1.5-Pro on a manually curated dataset of Basel III regulatory text.
• Implemented numerical simulations using fixed income, equities, currencies, and commodities to validate the mathematical accuracy of LLM-generated capital requirement calculations.
• Applied a troubleshooting process to identify and resolve parsing errors in mathematical formulas, legal terms, tables, and footnotes.
• Used a case study framework to test the end-to-end pipeline: document input → prompt processing → regulatory extraction → mathematical computation.

Figure 1 : Visual representation of the process for interpreting financial regulation documents.

Experimental results

Research questions

• RQ1 How do different LLMs (e.g., GPT-4, GPT-3.5, Claude-3, Gemini-1.5) perform in extracting and computing capital requirements from Basel III regulatory texts?
• RQ2 What is the impact of document loading format (PDF vs. image) on LLM accuracy in interpreting complex regulatory documents with mixed content (text, equations, tables, footnotes)?
• RQ3 To what extent does engineered prompt design improve LLM performance in identifying risk buckets, weights, and correlations compared to naive prompting?
• RQ4 Can LLMs accurately compute the minimum capital requirements under Basel III’s market risk framework when guided by structured prompts and optimal input formats?
• RQ5 What are the primary failure modes of LLMs in financial regulatory interpretation, and how can they be systematically mitigated?

Key findings

• GPT-4 achieved 100% accuracy in identifying risk weights and 85% accuracy in classifying risk buckets when using the detailed prompt, significantly outperforming the 65.5% accuracy of the naive prompt.
• GPT-4 achieved 96.5% accuracy in identifying correlations when documents were loaded as images, compared to 68% accuracy with PDFs, indicating superior performance with visual input.
• Claude-3-Opus achieved 97.5% accuracy with image loading, outperforming its 76.5% accuracy with PDFs, highlighting the advantage of image-based parsing for complex layouts.
• GPT-3.5 failed entirely in complex calculations, achieving 0% accuracy despite correctly identifying regulatory content, underscoring its limitations in mathematical reasoning.
• Gemini-1.5-Pro achieved 58% accuracy in correlation identification, demonstrating moderate performance but still falling short of GPT-4’s 96.5% with image input.
• The study confirms that prompt engineering and document format are critical factors—GPT-4’s performance improved from 68% to 96.5% in correlation detection when switching from PDF to image input.

Figure 2 : Detailed schematic of the Large Language Model used for financial document analysis.