Are Your LLMs Capable of Stable Reasoning?
The rapid advancement of Large Language Models (LLMs) has demonstrated
remarkable progress in complex reasoning tasks. However, a significant
discrepancy persists between benchmark performances and real-world
applications. We identify this gap as primarily stemming from current
evaluation protocols and metrics, which inadequately capture the full spectrum
of LLM capabilities, particularly in complex reasoning tasks where both
accuracy and consistency are crucial. This work makes two key contributions.
First, we introduce G-Pass@k, a novel evaluation metric that provides a
continuous assessment of model performance across multiple sampling attempts,
quantifying both the model's peak performance potential and its stability.
Second, we present LiveMathBench, a dynamic benchmark comprising challenging,
contemporary mathematical problems designed to minimize data leakage risks
during evaluation. Through extensive experiments using G-Pass@k on
state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights
into both their maximum capabilities and operational consistency. Our findings
reveal substantial room for improvement in LLMs' "realistic" reasoning
capabilities, highlighting the need for more robust evaluation methods. The
benchmark and detailed results are available at:
https://github.com/open-compass/GPassK.
67623bb9d44ba09e9119fe12
Junnan Liu
admin_assigned
2024-12-18T12:55:23.904Z
643d26979347842571bc9613
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3heFf7h3jbhhJWJ4JfGfh.jpeg
Junnan Liu
user
jnanliu
67623bb9d44ba09e9119fe13
Hongwei Liu
67623bb9d44ba09e9119fe14
Linchen Xiao
admin_assigned
2024-12-18T12:55:51.678Z
64f58f279eaf9d8fb746cd0e
/avatars/227db72e089026012694ff16cdf102d0.svg
LinchenXiao
user
LinchenXiao
67623bb9d44ba09e9119fe15
Ziyi Wang
67623bb9d44ba09e9119fe16
Kuikun Liu
claimed_verified
2024-12-18T09:52:14.336Z
63fd691794cc8f815d50c112
/avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
liu
user
Harold-lkk
67623bb9d44ba09e9119fe17
Songyang Gao
admin_assigned
2024-12-18T12:56:48.903Z
650ab54e23196fb2d86b486b
/avatars/e0506393589695b553ec9ee3fe99b93a.svg
SongYang Gao
user
Wizardcoast
67623bb9d44ba09e9119fe18
Wenwei Zhang
admin_assigned
2024-12-18T12:56:56.862Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
67623bb9d44ba09e9119fe19
Songyang Zhang
claimed_verified
2024-12-18T09:52:35.610Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
67623bb9d44ba09e9119fe1a
Kai Chen
Are Your LLMs Capable of Stable Reasoning?
The rapid advancement of Large Language Models (LLMs) has demonstrated
remarkable progress in complex reasoning tasks. However, a significant
discrepancy persists between benchmark performances and real-world
applications. We identify this gap as primarily stemming from current
evaluation protocols and metrics, which inadequately capture the full spectrum
of LLM capabilities, particularly in complex reasoning tasks where both
accuracy and consistency are crucial. This work makes two key contributions.
First, we introduce G-Pass@k, a novel evaluation metric that provides a
continuous assessment of model performance across multiple sampling attempts,
quantifying both the model's peak performance potential and its stability.
Second, we present LiveMathBench, a dynamic benchmark comprising challenging,
contemporary mathematical problems designed to minimize data leakage risks
during evaluation. Through extensive experiments using G-Pass@k on
state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights
into both their maximum capabilities and operational consistency. Our findings
reveal substantial room for improvement in LLMs' "realistic" reasoning
capabilities, highlighting the need for more robust evaluation methods. The
benchmark and detailed results are available at:
https://github.com/open-compass/GPassK.
G-Pass@k and LiveMathBench assess Large Language Models' reasoning capabilities and consistency using complex mathematical problems.
Large Language Models G-Pass@k LiveMathBench complex reasoning tasks sampling attempts data leakage evaluation metrics benchmark reasoning capabilities