UCFE: A User-Centric Financial Expertise Benchmark for Large Language
  Models
This paper introduces the UCFE: User-Centric Financial Expertise benchmark,
an innovative framework designed to evaluate the ability of large language
models (LLMs) to handle complex real-world financial tasks. UCFE benchmark
adopts a hybrid approach that combines human expert evaluations with dynamic,
task-specific interactions to simulate the complexities of evolving financial
scenarios. Firstly, we conducted a user study involving 804 participants,
collecting their feedback on financial tasks. Secondly, based on this feedback,
we created our dataset that encompasses a wide range of user intents and
interactions. This dataset serves as the foundation for benchmarking 12 LLM
services using the LLM-as-Judge methodology. Our results show a significant
alignment between benchmark scores and human preferences, with a Pearson
correlation coefficient of 0.78, confirming the effectiveness of the UCFE
dataset and our evaluation approach. UCFE benchmark not only reveals the
potential of LLMs in the financial sector but also provides a robust framework
for assessing their performance and user satisfaction.The benchmark dataset and
evaluation code are available.
6715fc2cac8cb216a41877a5
Yuzhe Yang
claimed_verified
2024-10-21T07:40:43.520Z
63f622c69cbd6730302783eb
https://cdn-avatars.huggingface.co/v1/production/uploads/63f622c69cbd6730302783eb/9cb96JVKiOm_JhF-shbFw.jpeg
Yuzhe Yang
user
TobyYang7
6715fc2cac8cb216a41877a6
Yifei Zhang
claimed_verified
2024-10-21T07:40:45.362Z
643c047326f177a3e41627b6
/avatars/ade75cebd049daf080ba80a80d516240.svg
Yifei Zhang
user
amstrongzyf
6715fc2cac8cb216a41877a7
Yan Hu
6715fc2cac8cb216a41877a8
Yilin Guo
6715fc2cac8cb216a41877a9
Ruoli Gan
admin_assigned
2024-10-21T12:04:36.004Z
651fdadbf998876ea7515265
/avatars/d43c4dc13764a0c43783c3b4e3fcda0e.svg
ganruoli
user
wittenberg
6715fc2cac8cb216a41877aa
Yueru He
admin_assigned
2024-10-21T12:04:42.030Z
65bd14e8ce846f8aa94db1d1
/avatars/76eaad15bf32eba75271f3dc315527c2.svg
Yueru He
user
Yueru1
6715fc2cac8cb216a41877ab
Mingcong Lei
admin_assigned
2024-10-21T12:05:00.431Z
6628c6107751d297d7025a71
https://cdn-avatars.huggingface.co/v1/production/uploads/6628c6107751d297d7025a71/S1rm5VIwV2Uxfv8GetKMU.jpeg
Lei Mingcong
user
SP4595
6715fc2cac8cb216a41877ac
Xiao Zhang
6715fc2cac8cb216a41877ad
Haining Wang
admin_assigned
2024-10-21T12:05:09.237Z
61f40f56b34a114b58db2fed
https://cdn-avatars.huggingface.co/v1/production/uploads/1643384634931-noauth.png
Haining Wang
user
haining
6715fc2cac8cb216a41877ae
Qianqian Xie
admin_assigned
2024-10-21T12:05:41.140Z
6479f4317c18dca75e9a9324
/avatars/9aa709230b057f57ee4415c04a622c63.svg
Xie
user
QianqianXie1994
6715fc2cac8cb216a41877af
Jimin Huang
admin_assigned
2024-10-21T12:06:03.105Z
63b58ed5889aa6707f0bb0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
Jimin Huang
user
jiminHuang
6715fc2cac8cb216a41877b0
Honghai Yu
6715fc2cac8cb216a41877b1
Benyou Wang
UCFE: A User-Centric Financial Expertise Benchmark for Large Language
  Models
This paper introduces the UCFE: User-Centric Financial Expertise benchmark,
an innovative framework designed to evaluate the ability of large language
models (LLMs) to handle complex real-world financial tasks. UCFE benchmark
adopts a hybrid approach that combines human expert evaluations with dynamic,
task-specific interactions to simulate the complexities of evolving financial
scenarios. Firstly, we conducted a user study involving 804 participants,
collecting their feedback on financial tasks. Secondly, based on this feedback,
we created our dataset that encompasses a wide range of user intents and
interactions. This dataset serves as the foundation for benchmarking 12 LLM
services using the LLM-as-Judge methodology. Our results show a significant
alignment between benchmark scores and human preferences, with a Pearson
correlation coefficient of 0.78, confirming the effectiveness of the UCFE
dataset and our evaluation approach. UCFE benchmark not only reveals the
potential of LLMs in the financial sector but also provides a robust framework
for assessing their performance and user satisfaction.The benchmark dataset and
evaluation code are available.
A user-centric framework evaluates LLMs in financial tasks through human feedback and task-specific interactions, achieving high correlation with human preferences.
Large language models (LLMs) LLM-as-Judge methodology User-Centric Financial Expertise (UCFE) hybrid approach human expert evaluations task-specific interactions user intentions benchmark scores Pearson correlation coefficient