[Paper Review] Financial Knowledge Large Language Model

Cehao Yang, Chengjin Xu+1 more|arXiv (Cornell University)|Jun 29, 2024

Stock Market Forecasting Methods5 citations

TL;DR

The paper proposes IDEA-FinBench for financial knowledge evaluation, IDEA-FinKER for rapid financial-domain adaptation of LLMs via retrieval-based and instruction-based knowledge injection, and IDEA-FinQA for real-time knowledge-enhanced financial question answering.

ABSTRACT

Artificial intelligence is making significant strides in the finance industry, revolutionizing how data is processed and interpreted. Among these technologies, large language models (LLMs) have demonstrated substantial potential to transform financial services by automating complex tasks, enhancing customer service, and providing detailed financial analysis. Firstly, we introduce IDEA-FinBench, an evaluation benchmark specifically tailored for assessing financial knowledge in large language models (LLMs). This benchmark utilizes questions from two globally respected and authoritative financial professional exams, aimimg to comprehensively evaluate the capability of LLMs to directly address exam questions pertinent to the finance sector. Secondly, we propose IDEA-FinKER, a Financial Knowledge Enhancement framework designed to facilitate the rapid adaptation of general LLMs to the financial domain, introducing a retrieval-based few-shot learning method for real-time context-level knowledge injection, and a set of high-quality financial knowledge instructions for fine-tuning any general LLM. Finally, we present IDEA-FinQA, a financial question-answering system powered by LLMs. This system is structured around a scheme of real-time knowledge injection and factual enhancement using external knowledge. IDEA-FinQA is comprised of three main modules: the data collector, the data querying module, and LLM-based agents tasked with specific functions.

Motivation & Objective

• Assess LLM financial knowledge using questions from CFA and CPA exams in Chinese and English across sixteen disciplines.
• Develop a retrieval-based few-shot framework to inject real-time financial context into LLMs (soft-injecting).
• Provide a hard-injection set of financial knowledge instructions to fine-tune general LLMs.
• Create a financial QA system (IDEA-FinQA) with real-time knowledge injection and modular data collection/querying.
• Offer an adaptable benchmark suite with modular evaluation and cross-linguistic prompt capabilities.

Proposed method

• Introduce IDEA-FinBench with CFA and CPA questions in Chinese/English across sixteen disciplines.
• Use embedding-based retrieval to support soft-injecting real-time context for LLMs (IDEA-FinKER).
• Provide a curated set of financial knowledge instructions for hard-injecting knowledge into LLMs.
• Design IDEA-FinQA with data collector, data querying module, and four LLM-based agents (query rewriter, intention detector, extractor/refiner, response generator).
• Leverage external knowledge sources and RAG to reduce hallucinations and improve factual grounding.

Experimental results

Research questions

• RQ1 Can LLMs directly answer CFA/CPA financial examination questions across multiple disciplines in Chinese and English?
• RQ2 Does a retrieval-based, few-shot knowledge injection improve financial reasoning and accuracy in LLMs?
• RQ3 Can a structured set of financial knowledge instructions (hard-injection) enhance domain-specific LLM performance?
• RQ4 Does real-time knowledge injection via external knowledge bases improve financial QA performance and reduce hallucinations?

Key findings

• IDEA-FinBench provides a benchmark grounded in CFA/CPA questions across 16 disciplines and supports modular, cross-linguistic evaluation.
• IDEA-FinKER substantially improves expert-level performance on IDEA-FinBench, especially for Chinese CPA-style questions.
• The soft-injecting retrieval-based paradigm enables real-time contextual knowledge enhancement for general LLMs.
• A hard-injecting paradigm with a financial knowledge instruction set supports fine-tuning for domain specialization.
• IDEA-FinQA demonstrates a three-module QA system leveraging real-time knowledge and external data to enhance factual accuracy.