The FinBen: An Holistic Financial Benchmark for Large Language Models
LLMs have transformed NLP and shown promise in various fields, yet their
potential in finance is underexplored due to a lack of thorough evaluations and
the complexity of financial tasks. This along with the rapid development of
LLMs, highlights the urgent need for a systematic financial evaluation
benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive
open-sourced evaluation benchmark, specifically designed to thoroughly assess
the capabilities of LLMs in the financial domain. FinBen encompasses 35
datasets across 23 financial tasks, organized into three spectrums of
difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs'
cognitive abilities in inductive reasoning, associative memory, quantitative
reasoning, crystallized intelligence, and more. Our evaluation of 15
representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals
insights into their strengths and limitations within the financial domain. The
findings indicate that GPT-4 leads in quantification, extraction, numerical
reasoning, and stock trading, while Gemini shines in generation and
forecasting; however, both struggle with complex extraction and forecasting,
showing a clear need for targeted enhancements. Instruction tuning boosts
simple task performance but falls short in improving complex reasoning and
forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance,
fostering AI development with regular updates of tasks and models.
65d56a82acacbc30a2eb627c
Qianqian Xie
claimed_verified
2024-03-25T09:44:31.521Z
6479f4317c18dca75e9a9324
/avatars/9aa709230b057f57ee4415c04a622c63.svg
Xie
user
QianqianXie1994
65d56a82acacbc30a2eb627d
Weiguang Han
admin_assigned
2024-02-21T12:03:28.892Z
6353a8e7eca5a823e3b578b6
/avatars/c533748aad297db0c0273c3b78f42ab4.svg
Weiguang Han
user
tothemoon96
65d56a82acacbc30a2eb627e
Zhengyu Chen
65d56a82acacbc30a2eb627f
Ruoyu Xiang
65d56a82acacbc30a2eb6280
Xiao Zhang
65d56a82acacbc30a2eb6281
Yueru He
admin_assigned
2024-02-21T12:04:49.547Z
65bd14e8ce846f8aa94db1d1
/avatars/76eaad15bf32eba75271f3dc315527c2.svg
Yueru He
user
Yueru1
65d56a82acacbc30a2eb6282
Mengxi Xiao
claimed_verified
2025-02-18T09:38:07.273Z
663adb42e14047f710dc1d29
/avatars/7ca49d67a4a8b4cf0ee896e07646715f.svg
Mengxi Xiao
user
ElsaShaw
65d56a82acacbc30a2eb6283
Dong Li
65d56a82acacbc30a2eb6284
Yongfu Dai
65d56a82acacbc30a2eb6285
Duanyu Feng
admin_assigned
2024-02-21T12:05:57.431Z
649d96407b4d8f568b49f472
/avatars/c0ca58462749108b5db9538b4be6d9ac.svg
Duanyu Feng
user
ColFeng
65d56a82acacbc30a2eb6286
Yijing Xu
65d56a82acacbc30a2eb6287
Haoqiang Kang
65d56a82acacbc30a2eb6288
Ziyan Kuang
65d56a82acacbc30a2eb6289
Chenhan Yuan
admin_assigned
2024-02-21T12:06:57.108Z
6447fbe54988ee01f2a59381
https://cdn-avatars.huggingface.co/v1/production/uploads/6447fbe54988ee01f2a59381/28zh9OsiMlB1ULVkIJTZh.jpeg
Chenhan Yuan
user
chrisyuan45
65d56a82acacbc30a2eb628a
Kailai Yang
admin_assigned
2024-02-21T12:07:04.217Z
646fc402e9c03ba436d5e93e
/avatars/870c86dc99fb1cb6a348a7a0385b1a04.svg
Kailai Yang
user
klyang
65d56a82acacbc30a2eb628b
Zheheng Luo
admin_assigned
2024-02-21T12:07:33.064Z
6443bb593c323e0918f61a96
/avatars/b9e1ba17f7798b5142bc0124fba95237.svg
zheheng luo
user
KenLuo
65d56a82acacbc30a2eb628c
Tianlin Zhang
65d56a82acacbc30a2eb628d
Zhiwei Liu
claimed_verified
2024-03-04T09:00:41.436Z
654d10d918c4ae0b4017fa4a
/avatars/a905ef9419ecffab40b17bdfcf23dce2.svg
Zhiwei Liu
user
lzw1008
65d56a82acacbc30a2eb628e
Guojun Xiong
65d56a82acacbc30a2eb628f
Zhiyang Deng
65d56a82acacbc30a2eb6290
Yuechen Jiang
65d56a82acacbc30a2eb6291
Zhiyuan Yao
65d56a82acacbc30a2eb6292
Haohang Li
65d56a82acacbc30a2eb6293
Yangyang Yu
65d56a82acacbc30a2eb6294
Gang Hu
65d56a82acacbc30a2eb6295
Jiajia Huang
65d56a82acacbc30a2eb6296
Xiao-Yang Liu
65d56a82acacbc30a2eb6297
Alejandro Lopez-Lira
admin_assigned
2024-02-21T12:10:25.133Z
65b473e5e602b6c2c92cbc8e
/avatars/a80556ea72f19289be49321f44f0c43a.svg
Alejandro Lopez-Lira
user
alejandroll10
65d56a82acacbc30a2eb6298
Benyou Wang
admin_assigned
2024-02-21T12:11:09.250Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
65d56a82acacbc30a2eb6299
Yanzhao Lai
65d56a82acacbc30a2eb629a
Hao Wang
65d56a82acacbc30a2eb629b
Min Peng
65d56a82acacbc30a2eb629c
Sophia Ananiadou
65d56a82acacbc30a2eb629d
Jimin Huang
admin_assigned
2024-02-21T12:11:48.539Z
63b58ed5889aa6707f0bb0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
Jimin Huang
user
jiminHuang
The FinBen: An Holistic Financial Benchmark for Large Language Models
LLMs have transformed NLP and shown promise in various fields, yet their
potential in finance is underexplored due to a lack of thorough evaluations and
the complexity of financial tasks. This along with the rapid development of
LLMs, highlights the urgent need for a systematic financial evaluation
benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive
open-sourced evaluation benchmark, specifically designed to thoroughly assess
the capabilities of LLMs in the financial domain. FinBen encompasses 35
datasets across 23 financial tasks, organized into three spectrums of
difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs'
cognitive abilities in inductive reasoning, associative memory, quantitative
reasoning, crystallized intelligence, and more. Our evaluation of 15
representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals
insights into their strengths and limitations within the financial domain. The
findings indicate that GPT-4 leads in quantification, extraction, numerical
reasoning, and stock trading, while Gemini shines in generation and
forecasting; however, both struggle with complex extraction and forecasting,
showing a clear need for targeted enhancements. Instruction tuning boosts
simple task performance but falls short in improving complex reasoning and
forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance,
fostering AI development with regular updates of tasks and models.
FinBen is an open-sourced financial evaluation benchmark for LLMs, assessing their capabilities across various tasks and difficulty levels, revealing strengths and weaknesses for targeted enhancements.
LLMs NLP FinBen evaluation benchmark Cattell-Horn-Carroll theory inductive reasoning associative memory quantitative reasoning crystallized intelligence GPT-4 ChatGPT Gemini instruction tuning complex reasoning forecasting