[Paper Review] BizFinBench.v2: A Unified Dual-Mode Bilingual Benchmark for Expert-Level Financial Capability Alignment

Xin Guo, Rongjunchen Zhang+5 more|arXiv (Cornell University)|Jan 10, 2026

Stock Market Forecasting Methods0 citations

TL;DR

BizFinBench.v2 is a large-scale bilingual benchmark using authentic Chinese and U.S. market data with offline and online tasks to evaluate LLMs’ real-world financial capabilities.

ABSTRACT

Large language models have undergone rapid evolution, emerging as a pivotal technology for intelligence in financial operations. However, existing benchmarks are often constrained by pitfalls such as reliance on simulated or general-purpose samples and a focus on singular, offline static scenarios. Consequently, they fail to align with the requirements for authenticity and real-time responsiveness in financial services, leading to a significant discrepancy between benchmark performance and actual operational efficacy. To address this, we introduce BizFinBench.v2, the first large-scale evaluation benchmark grounded in authentic business data from both Chinese and U.S. equity markets, integrating online assessment. We performed clustering analysis on authentic user queries from financial platforms, resulting in eight fundamental tasks and two online tasks across four core business scenarios, totaling 29,578 expert-level Q&A pairs. Experimental results demonstrate that ChatGPT-5 achieves a prominent 61.5% accuracy in main tasks, though a substantial gap relative to financial experts persists; in online tasks, DeepSeek-R1 outperforms all other commercial LLMs. Error analysis further identifies the specific capability deficiencies of existing models within practical financial business contexts. BizFinBench.v2 transcends the limitations of current benchmarks, achieving a business-level deconstruction of LLM financial capabilities and providing a precise basis for evaluating efficacy in the widespread deployment of LLMs within the financial domain. The data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.

Motivation & Objective

• Capture authentic financial business capabilities using real-world Chinese and U.S. market data.
• Bridge the gap between offline benchmarks and online, real-time financial service needs.
• Provide a dual-track evaluation framework: Core Business Capabilities + Online Performance.
• Identify core capability deficiencies of LLMs through expert-informed error analysis.

Proposed method

• Construct eight offline tasks and two online tasks across four core business scenarios from real market data.
• Organize tasks into Core Business Capabilities and Online Performance for dual-track evaluation.
• Apply rigorous three-level quality control (platform clustering, frontline review, expert cross-validation) to ensure data quality and compliance.
• Use real-time online data for Stock Price Prediction and Portfolio Asset Allocation tasks.
• Evaluate 21 LLMs (proprietary and open-source) in zero-shot settings, with conformal prediction for SA and SPP tasks.
• Provide an open-source LLM investment system for reproducible online evaluation in PAA.

Figure 1: BizFinBench.v2 comprises eight foundational tasks and two online tasks distributed across four major scenarios. The top-right corner displays a real-time screenshot of the Portfolio Asset Allocation task.

Experimental results

Research questions

• RQ1 How well do LLMs perform on authentic financial tasks drawn from Chinese and U.S. markets in offline settings?
• RQ2 How capable are LLMs at online, real-time financial tasks such as stock price prediction and asset allocation?
• RQ3 What are the common error modes of LLMs in practical financial business contexts and how can they be mitigated?
• RQ4 How does performance vary between general-purpose and finance-specialized LLMs on real-world financial data?

Key findings

• ChatGPT-5 achieves the highest average accuracy (61.5%) across main offline tasks.
• In online tasks, DeepSeek-R1 surpasses all other commercial LLMs.
• Open-source model Qwen3-235B-A22B-Thinking-2507 tops open-source results with 53.3% average accuracy.
• Financial experts achieve higher baseline capability (84.8%) than current LLMs in foundational tasks.
• Error analysis reveals five business dilemmas: Financial Semantic Deviation, Long-term Business Logic Discontinuity, MIAD, High-precision Computational Distortion, and Financial Time-Series Logical Disorder.
• DeepSeek-R1 excels in asset allocation metrics (e.g., total return, Sharpe ratio) among commercial models; some top models struggle to beat SPY benchmarks in this task.

Figure 2: We have ranked the performance of the LLMs participating in the evaluation under the zero-shot setting, and these results reflect their authentic practical business capabilities.