[Paper Review] FinBen: A Holistic Financial Benchmark for Large Language Models

Qianqian Xie, Weiguang Han+32 more|arXiv (Cornell University)|Feb 20, 2024

FinTech, Crowdfunding, Digital Finance21 citations

TL;DR

FinBen introduces a comprehensive open-source benchmark with 35 datasets across 23 financial tasks, organized into three CHC-inspired spectrums to evaluate LLMs from induction to trading.

ABSTRACT

LLMs have transformed NLP and shown promise in various fields, yet their potential in finance is underexplored due to a lack of comprehensive evaluation benchmarks, the rapid development of LLMs, and the complexity of financial tasks. In this paper, we introduce FinBen, the first extensive open-source evaluation benchmark, including 36 datasets spanning 24 financial tasks, covering seven critical aspects: information extraction (IE), textual analysis, question answering (QA), text generation, risk management, forecasting, and decision-making. FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and three novel open-source evaluation datasets for text summarization, question answering, and stock trading. Our evaluation of 15 representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals several key findings: While LLMs excel in IE and textual analysis, they struggle with advanced reasoning and complex tasks like text generation and forecasting. GPT-4 excels in IE and stock trading, while Gemini is better at text generation and forecasting. Instruction-tuned LLMs improve textual analysis but offer limited benefits for complex tasks such as QA. FinBen has been used to host the first financial LLMs shared task at the FinNLP-AgentScen workshop during IJCAI-2024, attracting 12 teams. Their novel solutions outperformed GPT-4, showcasing FinBen's potential to drive innovation in financial LLMs. All datasets, results, and codes are released for the research community: https://github.com/The-FinAI/PIXIU.

Motivation & Objective

• Motivate the need for a broad, real-world financial evaluation benchmark for LLMs.
• Design FinBen to cover linguistic, knowledge extraction, numerical reasoning, generation, forecasting, and trading tasks.
• Provide an open-source framework with diverse data modalities to gauge LLM capabilities in finance.
• Assess 15 representative LLMs to identify strengths and limitations across financial tasks.
• Propose CHC-inspired spectrums to map cognitive abilities from basic to general intelligence in finance.

Proposed method

• Construct FinBen with 35 datasets across 23 financial tasks.
• Organize tasks into three spectrums mirroring CHC theory: Spectrum I (Quantification, Extraction, Numerical Understanding), Spectrum II (Generation, Forecasting), Spectrum III (Stock Trading).
• Evaluate zero-shot and few-shot performance of 15 LLMs, including GPT-4, ChatGPT, Gemini, and open-source models.
• Use standard metrics per task (e.g., F1, accuracy, RMSE, ROUGE/BERTScore/BARTScore, MCC, EMAcc) and trading metrics (CR, SR, DV, AV, MD).
• Compare performance across tasks to identify where instruction tuning helps and where gaps remain.

Experimental results

Research questions

• RQ1 Can FinBen provide a broad, real-world assessment of LLMs in finance beyond existing NLP-centric benchmarks?
• RQ2 Which financial tasks do current LLMs excel at, and where do they struggle (e.g., complex extraction, numerical reasoning, forecasting)?
• RQ3 How do different model families (GPT-4, Gemini, open-source LLMs) compare across the three CHC-inspired spectrums?
• RQ4 Does instruction tuning improve performance uniformly across tasks, or only on simpler ones?

Key findings

• GPT-4 leads in quantification, extraction, numerical reasoning, and stock trading; Gemini excels in generation and forecasting.
• Instruction tuning boosts simple tasks but is less effective for complex numerical reasoning, generation, and forecasting.
• Open-source/Chinese-tuned models show strong performance on some classification tasks, but cross-lingual effects and dataset alignment influence results.
• Stock trading tasks reveal general intelligence capabilities in LLMs, with GPT-4 achieving the highest Sharpe Ratio and the lowest max drawdown among evaluated models.