MultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark
  for Financial LLM Evaluation
Recent advances in large language models (LLMs) have accelerated progress in
financial NLP and applications, yet existing benchmarks remain limited to
monolingual and unimodal settings, often over-relying on simple tasks and
failing to reflect the complexity of real-world financial communication. We
introduce MultiFinBen, the first multilingual and multimodal benchmark tailored
to the global financial domain, evaluating LLMs across modalities (text,
vision, audio) and linguistic settings (monolingual, bilingual, multilingual)
on domain-specific tasks. We introduce two novel tasks, including PolyFiQA-Easy
and PolyFiQA-Expert, the first multilingual financial benchmarks requiring
models to perform complex reasoning over mixed-language inputs; and EnglishOCR
and SpanishOCR, the first OCR-embedded financial QA tasks challenging models to
extract and reason over information from visual-text financial documents.
Moreover, we propose a dynamic, difficulty-aware selection mechanism and curate
a compact, balanced benchmark rather than simple aggregation existing datasets.
Extensive evaluation of 22 state-of-the-art models reveals that even the
strongest models, despite their general multimodal and multilingual
capabilities, struggle dramatically when faced with complex cross-lingual and
multimodal tasks in financial domain. MultiFinBen is publicly released to
foster transparent, reproducible, and inclusive progress in financial studies
and applications.
685240aa0164cd131671056a
Xueqing Peng
claimed_verified
2025-06-18T16:15:27.517Z
63a0c0803c8841cfe2cd1f15
/avatars/bbe216db7a33612f23d23ce4ed4ba3f9.svg
Xueqing Peng
user
Xueqing
685240aa0164cd131671056b
Lingfei Qian
685240aa0164cd131671056c
Yan Wang
claimed_verified
2025-06-18T16:15:31.114Z
65d76cc5b9b7b8bf88faa916
/avatars/d95232cd0c307efab6197ade1a66190b.svg
Yan Wang
user
YanAdjeNole
685240aa0164cd131671056d
Ruoyu Xiang
685240aa0164cd131671056e
Yueru He
claimed_verified
2025-06-18T16:15:33.127Z
65bd14e8ce846f8aa94db1d1
/avatars/76eaad15bf32eba75271f3dc315527c2.svg
Yueru He
user
Yueru1
685240aa0164cd131671056f
Yang Ren
685240aa0164cd1316710570
Mingyang Jiang
claimed_verified
2025-06-18T16:15:29.378Z
678ab76d27bb31ad067cbffd
https://cdn-avatars.huggingface.co/v1/production/uploads/678ab76d27bb31ad067cbffd/l_5mRG6_BmqmADKs2Kb3W.png
Mingyang Jiang
user
0oJ1mmyo0
685240aa0164cd1316710571
Jeff Zhao
685240aa0164cd1316710572
Huan He
685240aa0164cd1316710573
Yi Han
685240aa0164cd1316710574
Yun Feng
685240aa0164cd1316710575
Yuechen Jiang
685240aa0164cd1316710576
Yupeng Cao
685240aa0164cd1316710577
Haohang Li
685240aa0164cd1316710578
Yangyang Yu
685240aa0164cd1316710579
Xiaoyu Wang
685240aa0164cd131671057a
Penglei Gao
685240aa0164cd131671057b
Shengyuan Lin
685240aa0164cd131671057c
Keyi Wang
685240aa0164cd131671057d
Shanshan Yang
685240aa0164cd131671057e
Yilun Zhao
claimed_verified
2025-06-18T10:58:59.111Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
685240aa0164cd131671057f
Zhiwei Liu
685240aa0164cd1316710580
Peng Lu
685240aa0164cd1316710581
Jerry Huang
685240aa0164cd1316710582
Suyuchen Wang
claimed_verified
2025-06-18T16:15:35.878Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/MWNanci3x5g780xh-704U.png
Suyuchen Wang
user
sheryc
685240aa0164cd1316710583
Triantafillos Papadopoulos
685240aa0164cd1316710584
Polydoros Giannouris
685240aa0164cd1316710585
Efstathia Soufleri
685240aa0164cd1316710586
Nuo Chen
685240aa0164cd1316710587
Guojun Xiong
685240aa0164cd1316710588
Zhiyang Deng
685240aa0164cd1316710589
Yijia Zhao
685240aa0164cd131671058a
Mingquan Lin
685240aa0164cd131671058b
Meikang Qiu
685240aa0164cd131671058c
Kaleb E Smith
685240aa0164cd131671058d
Arman Cohan
685240aa0164cd131671058e
Xiao-Yang Liu
685240aa0164cd131671058f
Jimin Huang
claimed_verified
2025-06-19T09:10:29.859Z
63b58ed5889aa6707f0bb0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
Jimin Huang
user
jiminHuang
685240aa0164cd1316710590
Alejandro Lopez-Lira
685240aa0164cd1316710591
Xi Chen
685240aa0164cd1316710592
Junichi Tsujii
685240aa0164cd1316710593
Jian-Yun Nie
685240aa0164cd1316710594
Sophia Ananiadou
685240aa0164cd1316710595
Qianqian Xie
MultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark
  for Financial LLM Evaluation
Recent advances in large language models (LLMs) have accelerated progress in
financial NLP and applications, yet existing benchmarks remain limited to
monolingual and unimodal settings, often over-relying on simple tasks and
failing to reflect the complexity of real-world financial communication. We
introduce MultiFinBen, the first multilingual and multimodal benchmark tailored
to the global financial domain, evaluating LLMs across modalities (text,
vision, audio) and linguistic settings (monolingual, bilingual, multilingual)
on domain-specific tasks. We introduce two novel tasks, including PolyFiQA-Easy
and PolyFiQA-Expert, the first multilingual financial benchmarks requiring
models to perform complex reasoning over mixed-language inputs; and EnglishOCR
and SpanishOCR, the first OCR-embedded financial QA tasks challenging models to
extract and reason over information from visual-text financial documents.
Moreover, we propose a dynamic, difficulty-aware selection mechanism and curate
a compact, balanced benchmark rather than simple aggregation existing datasets.
Extensive evaluation of 22 state-of-the-art models reveals that even the
strongest models, despite their general multimodal and multilingual
capabilities, struggle dramatically when faced with complex cross-lingual and
multimodal tasks in financial domain. MultiFinBen is publicly released to
foster transparent, reproducible, and inclusive progress in financial studies
and applications.
MultiFinBen is a multilingual and multimodal benchmark for financial domain tasks, evaluating LLMs across modalities and linguistic settings, revealing challenges in complex cross-lingual and multimodal financial reasoning.
LLMs financial NLP multilingual multimodal benchmark domain-specific tasks PolyFiQA-Easy PolyFiQA-Expert EnglishOCR SpanishOCR dynamic selection mechanism difficulty-aware OCR-embedded financial QA