BizFinBench: A Business-Driven Real-World Financial Benchmark for
  Evaluating LLMs
Large language models excel in general tasks, yet assessing their reliability
in logic-heavy, precision-critical domains like finance, law, and healthcare
remains challenging. To address this, we introduce BizFinBench, the first
benchmark specifically designed to evaluate LLMs in real-world financial
applications. BizFinBench consists of 6,781 well-annotated queries in Chinese,
spanning five dimensions: numerical calculation, reasoning, information
extraction, prediction recognition, and knowledge-based question answering,
grouped into nine fine-grained categories. The benchmark includes both
objective and subjective metrics. We also introduce IteraJudge, a novel LLM
evaluation method that reduces bias when LLMs serve as evaluators in objective
metrics. We benchmark 25 models, including both proprietary and open-source
systems. Extensive experiments show that no model dominates across all tasks.
Our evaluation reveals distinct capability patterns: (1) In Numerical
Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while
smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning,
proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with
open-source models trailing by up to 19.49 points; (3) In Information
Extraction, the performance spread is the largest, with DeepSeek-R1 scoring
71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition,
performance variance is minimal, with top models scoring between 39.16 and
50.00. We find that while current LLMs handle routine finance queries
competently, they struggle with complex scenarios requiring cross-concept
reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future
research. The code and dataset are available at
https://github.com/HiThink-Research/BizFinBench.
683536ec70d215849adfc236
Guilong Lu
admin_assigned
2025-05-27T08:23:29.797Z
6440f70f1a80f6d83cadfd16
/avatars/04790922837dac81747e80bd0ee0a1cf.svg
luguilong
user
guilong
683536ec70d215849adfc237
Xuntao Guo
admin_assigned
2025-05-27T08:23:39.616Z
672b138db4215fd3888e0a8f
/avatars/e90fe671a1db66401db88429fae9a763.svg
guo
user
xuntao
683536ec70d215849adfc238
Rongjunchen Zhang
extracted_pending
2025-05-27T03:52:17.018Z
6555df426947208b7741b637
https://cdn-avatars.huggingface.co/v1/production/uploads/6555df426947208b7741b637/b7ply-HyaPKXrPvRNh21K.jpeg
Rongjunchen Zhang
user
Tinker250
683536ec70d215849adfc239
Wenqiao Zhu
claimed_verified
2025-05-27T07:49:28.216Z
648add6aff6123185eb185a8
/avatars/e37dfa680c1bb86c721165f03eb79e97.svg
WNQzhu
user
Qlisp
683536ec70d215849adfc23a
Ji Liu
BizFinBench: A Business-Driven Real-World Financial Benchmark for
  Evaluating LLMs
Large language models excel in general tasks, yet assessing their reliability
in logic-heavy, precision-critical domains like finance, law, and healthcare
remains challenging. To address this, we introduce BizFinBench, the first
benchmark specifically designed to evaluate LLMs in real-world financial
applications. BizFinBench consists of 6,781 well-annotated queries in Chinese,
spanning five dimensions: numerical calculation, reasoning, information
extraction, prediction recognition, and knowledge-based question answering,
grouped into nine fine-grained categories. The benchmark includes both
objective and subjective metrics. We also introduce IteraJudge, a novel LLM
evaluation method that reduces bias when LLMs serve as evaluators in objective
metrics. We benchmark 25 models, including both proprietary and open-source
systems. Extensive experiments show that no model dominates across all tasks.
Our evaluation reveals distinct capability patterns: (1) In Numerical
Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while
smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning,
proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with
open-source models trailing by up to 19.49 points; (3) In Information
Extraction, the performance spread is the largest, with DeepSeek-R1 scoring
71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition,
performance variance is minimal, with top models scoring between 39.16 and
50.00. We find that while current LLMs handle routine finance queries
competently, they struggle with complex scenarios requiring cross-concept
reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future
research. The code and dataset are available at
https://github.com/HiThink-Research/BizFinBench.
BizFinBench is a benchmark for evaluating large language models in financial applications, revealing distinct performance patterns across various tasks.
large language models BizFinBench numerical calculation reasoning information extraction prediction recognition knowledge-based question answering IteraJudge Claude-3.5-Sonnet DeepSeek-R1 Qwen2.5-VL-3B ChatGPT-o3 Gemini-2.0-Flash Qwen3-1.7B