A Survey on Large Language Model Benchmarks
In recent years, with the rapid development of the depth and breadth of large
language models' capabilities, various corresponding evaluation benchmarks have
been emerging in increasing numbers. As a quantitative assessment tool for
model performance, benchmarks are not only a core means to measure model
capabilities but also a key element in guiding the direction of model
development and promoting technological innovation. We systematically review
the current status and development of large language model benchmarks for the
first time, categorizing 283 representative benchmarks into three categories:
general capabilities, domain-specific, and target-specific. General capability
benchmarks cover aspects such as core linguistics, knowledge, and reasoning;
domain-specific benchmarks focus on fields like natural sciences, humanities
and social sciences, and engineering technology; target-specific benchmarks pay
attention to risks, reliability, agents, etc. We point out that current
benchmarks have problems such as inflated scores caused by data contamination,
unfair evaluation due to cultural and linguistic biases, and lack of evaluation
on process credibility and dynamic environments, and provide a referable design
paradigm for future benchmark innovation.
68a7f2a139413c456c05b17c
Shiwen Ni
admin_assigned
2025-08-22T13:05:50.349Z
641944acf9d6f1d772ede902
/avatars/a2f88017f7ab0cbea6862201e35fe747.svg
ShiwenNi
user
ShiwenNi
68a7f2a139413c456c05b17d
Guhong Chen
admin_assigned
2025-08-22T13:06:02.846Z
66c2bdaa5bdd611f9a615336
/avatars/5356e6560f9fd84b43bce1b990275f92.svg
Guhong Chen
user
youzi517
68a7f2a139413c456c05b17e
Shuaimin Li
admin_assigned
2025-08-22T13:06:08.624Z
67da79d94114b77b63ff5787
/avatars/6d173cc99c5555fe9054ba9f770003db.svg
shuaimin li
user
shuaiminli
68a7f2a139413c456c05b17f
Xuanang Chen
admin_assigned
2025-08-22T13:05:57.125Z
63ef664304b0e373992a2633
/avatars/cba554ff88bd8b68ae51bea8ee991d13.svg
Xuanang Chen
user
xuanang
68a7f2a139413c456c05b180
Siyi Li
68a7f2a139413c456c05b181
Bingli Wang
claimed_verified
2025-08-25T08:21:36.132Z
658d0a228cff48d3a4612689
/avatars/70e297c6cb12d1bdde6d91c23f590b63.svg
Bingli Wang
user
BingliW
68a7f2a139413c456c05b182
Qiyao Wang
claimed_verified
2025-08-25T08:27:18.014Z
64560618bfdf9c63ce2d658a
https://cdn-avatars.huggingface.co/v1/production/uploads/64560618bfdf9c63ce2d658a/GVBWU4yNzRsjdyzKT3z3B.jpeg
Mathsion Wong
user
QiYao-Wang
68a7f2a139413c456c05b183
Xingjian Wang
68a7f2a139413c456c05b184
Yifan Zhang
68a7f2a139413c456c05b185
Liyang Fan
admin_assigned
2025-08-22T13:06:41.019Z
6601302c7a3a9bbfcd960f77
/avatars/cc374ce5e192cb5817fc34f09d91a49a.svg
Liyang Fan
user
Relic-Yuexi
68a7f2a139413c456c05b186
Chengming Li
68a7f2a139413c456c05b187
Ruifeng Xu
68a7f2a139413c456c05b188
Le Sun
68a7f2a139413c456c05b189
Min Yang
A Survey on Large Language Model Benchmarks
In recent years, with the rapid development of the depth and breadth of large
language models' capabilities, various corresponding evaluation benchmarks have
been emerging in increasing numbers. As a quantitative assessment tool for
model performance, benchmarks are not only a core means to measure model
capabilities but also a key element in guiding the direction of model
development and promoting technological innovation. We systematically review
the current status and development of large language model benchmarks for the
first time, categorizing 283 representative benchmarks into three categories:
general capabilities, domain-specific, and target-specific. General capability
benchmarks cover aspects such as core linguistics, knowledge, and reasoning;
domain-specific benchmarks focus on fields like natural sciences, humanities
and social sciences, and engineering technology; target-specific benchmarks pay
attention to risks, reliability, agents, etc. We point out that current
benchmarks have problems such as inflated scores caused by data contamination,
unfair evaluation due to cultural and linguistic biases, and lack of evaluation
on process credibility and dynamic environments, and provide a referable design
paradigm for future benchmark innovation.
A systematic review of large language model benchmarks identifies issues such as data contamination, cultural biases, and lack of process credibility, and proposes a design paradigm for future improvements.
large language models evaluation benchmarks general capabilities domain-specific target-specific core linguistics knowledge reasoning natural sciences humanities social sciences engineering technology data contamination cultural biases process credibility dynamic environments