Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning
  Capabilities Through Evaluation Design
Reasoning models represented by the Deepseek-R1-Distill series have been
widely adopted by the open-source community due to their strong performance in
mathematics, science, programming, and other domains. However, our study
reveals that their benchmark evaluation results are subject to significant
fluctuations caused by various factors. Subtle differences in evaluation
conditions can lead to substantial variations in results. Similar phenomena are
observed in other open-source inference models fine-tuned based on the
Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their
claimed performance improvements difficult to reproduce reliably. Therefore, we
advocate for the establishment of a more rigorous paradigm for model
performance evaluation and present our empirical assessments of the
Deepseek-R1-Distill series models.
6842537f1c4f28a2031f499c
Lin Sun
claimed_verified
2025-06-06T07:41:08.211Z
632c30576bcb864974cc40a8
/avatars/96aa948ad1dd35d355e20b5765a2563a.svg
sunlin
user
lincharliesun
6842537f1c4f28a2031f499d
Weihong Lin
6842537f1c4f28a2031f499e
Jinzhu Wu
6842537f1c4f28a2031f499f
Yongfu Zhu
claimed_verified
2025-06-10T09:22:29.244Z
668cf9d4f7c08c3f9e12f97b
/avatars/396be7f144784aa2a1c35ec059a84997.svg
JosephZhu
user
JjjjjZzz
6842537f1c4f28a2031f49a0
Xiaoqi Jian
claimed_verified
2025-06-07T05:48:55.525Z
64be2455b567ae97c34bb948
https://cdn-avatars.huggingface.co/v1/production/uploads/64be2455b567ae97c34bb948/QuCdStDDGaXjDmp4V-dBj.jpeg
Xiaoqi Jian
user
mx1024
6842537f1c4f28a2031f49a1
Guangxiang Zhao
6842537f1c4f28a2031f49a2
Change Jia
6842537f1c4f28a2031f49a3
Linglin Zhang
6842537f1c4f28a2031f49a4
Sai-er Hu
6842537f1c4f28a2031f49a5
Yuhan Wu
6842537f1c4f28a2031f49a6
Xiangzheng Zhang
Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning
  Capabilities Through Evaluation Design
Reasoning models represented by the Deepseek-R1-Distill series have been
widely adopted by the open-source community due to their strong performance in
mathematics, science, programming, and other domains. However, our study
reveals that their benchmark evaluation results are subject to significant
fluctuations caused by various factors. Subtle differences in evaluation
conditions can lead to substantial variations in results. Similar phenomena are
observed in other open-source inference models fine-tuned based on the
Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their
claimed performance improvements difficult to reproduce reliably. Therefore, we
advocate for the establishment of a more rigorous paradigm for model
performance evaluation and present our empirical assessments of the
Deepseek-R1-Distill series models.
Empirical assessments reveal significant fluctuations in benchmark evaluation results of Deepseek-R1-Distill models, questioning the reliability of claimed performance improvements and advocating for a more rigorous evaluation paradigm.
reasoning models Deepseek-R1-Distill benchmark evaluation open-source inference models performance variations QwQ-32B model