FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
Multimodal Large Language Models (MLLMs) have experienced rapid development
in recent years. However, in the financial domain, there is a notable lack of
effective and specialized multimodal evaluation datasets. To advance the
development of MLLMs in the finance domain, we introduce FinMME, encompassing
more than 11,000 high-quality financial research samples across 18 financial
domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We
ensure data quality through 20 annotators and carefully designed validation
mechanisms. Additionally, we develop FinScore, an evaluation system
incorporating hallucination penalties and multi-dimensional capability
assessment to provide an unbiased evaluation. Extensive experimental results
demonstrate that even state-of-the-art models like GPT-4o exhibit
unsatisfactory performance on FinMME, highlighting its challenging nature. The
benchmark exhibits high robustness with prediction variations under different
prompts remaining below 1%, demonstrating superior reliability compared to
existing datasets. Our dataset and evaluation protocol are available at
https://huggingface.co/datasets/luojunyu/FinMME and
https://github.com/luo-junyu/FinMME.
683d04c751706d12b2c262ea
Junyu Luo
claimed_verified
2025-06-04T09:03:26.483Z
642da1cd99f3110ac27caca5
https://cdn-avatars.huggingface.co/v1/production/uploads/642da1cd99f3110ac27caca5/C1QJY3R_ZdaeANG1y8iW7.jpeg
junyu
user
luojunyu
683d04c751706d12b2c262eb
Zhizhuo Kou
claimed_verified
2025-06-04T09:03:22.133Z
65a9c8652bf3e0cbbfcab2c8
/avatars/fc690a78b5f2e94e08a40059ae40625c.svg
Alan KOU
user
alan1027
683d04c751706d12b2c262ec
Liming Yang
claimed_verified
2025-06-04T09:03:24.346Z
6434c115a5aed21dd11981c5
/avatars/d51e6e384cfc3affe578e7816bcebb35.svg
Yang Liming
user
chunfenri
683d04c751706d12b2c262ed
Xiao Luo
683d04c751706d12b2c262ee
Jinsheng Huang
683d04c751706d12b2c262ef
Zhiping Xiao
683d04c751706d12b2c262f0
Jingshu Peng
683d04c751706d12b2c262f1
Chengzhong Liu
683d04c751706d12b2c262f2
Jiaming Ji
683d04c751706d12b2c262f3
Xuanzhe Liu
683d04c751706d12b2c262f4
Sirui Han
683d04c751706d12b2c262f5
Ming Zhang
683d04c751706d12b2c262f6
Yike Guo
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
Multimodal Large Language Models (MLLMs) have experienced rapid development
in recent years. However, in the financial domain, there is a notable lack of
effective and specialized multimodal evaluation datasets. To advance the
development of MLLMs in the finance domain, we introduce FinMME, encompassing
more than 11,000 high-quality financial research samples across 18 financial
domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We
ensure data quality through 20 annotators and carefully designed validation
mechanisms. Additionally, we develop FinScore, an evaluation system
incorporating hallucination penalties and multi-dimensional capability
assessment to provide an unbiased evaluation. Extensive experimental results
demonstrate that even state-of-the-art models like GPT-4o exhibit
unsatisfactory performance on FinMME, highlighting its challenging nature. The
benchmark exhibits high robustness with prediction variations under different
prompts remaining below 1%, demonstrating superior reliability compared to
existing datasets. Our dataset and evaluation protocol are available at
https://huggingface.co/datasets/luojunyu/FinMME and
https://github.com/luo-junyu/FinMME.
FinMME is a comprehensive multimodal dataset for financial research and FinScore is an evaluation system that highlights the challenges faced by even advanced models like GPT-4o in the finance domain.
Multimodal Large Language Models FinMME financial research samples high-quality dataset financial domains asset classes chart types data quality FinScore hallucination penalties multi-dimensional capability assessment benchmark dataset prediction robustness