CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End
  Code Review Evaluation in Python Projects
Automated code review (CR) is a key application for Large Language Models
(LLMs), but progress is hampered by a "reality gap": existing benchmarks
evaluate models on isolated sub-tasks using simplified, context-poor data. This
fails to reflect the holistic context-rich nature of real-world CR. To bridge
this gap, we introduce CodeFuse-CR-Bench, the first comprehensiveness-aware
benchmark for repository-level CR evaluation. CodeFuse-CR-Bench comprises 601
high-quality instances from 70 Python projects covering nine Pull-Request (PR)
problem domains, where each instance provides rich, multi-faceted context
including the associated issue, PR details, and repository state, enabling
end-to-end evaluation. Beyond superficial metrics, we also propose a novel
evaluation framework that combines rule-based checks for location and syntax
with model-based judgments of review quality. We present the first large-scale
assessment of state-of-the-art LLMs on this comprehensive CR task. Our results
establish crucial baselines and reveal that (1) no single LLM dominates all
aspects of CR; (2) Gemini 2.5 Pro achieves the highest comprehensive
performance; and (3) different LLMs exhibit varying robustness to redundant
context. These findings highlight the necessity of holistic, multi-dimensional
evaluation and provide actionable insights for advancing truly intelligent yet
practical CR assistants.
68d2371c1ca7156988a8ee2e
Hanyang Guo
68d2371c1ca7156988a8ee2f
Xunjin Zheng
68d2371c1ca7156988a8ee30
Zihan Liao
68d2371c1ca7156988a8ee31
Hang Yu
claimed_verified
2025-10-10T09:59:39.984Z
6837243353739ee0d588d04c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CYU8xxEHxxGkwNVCpua6X.png
Hang Yu
user
fhlyhv
68d2371c1ca7156988a8ee32
Peng DI
68d2371c1ca7156988a8ee33
Ziyin Zhang
claimed_verified
2025-09-23T10:05:07.145Z
6430bdd8cd31d174a9f900fb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
Ziyin Zhang
user
Geralt-Targaryen
68d2371c1ca7156988a8ee34
Hong-Ning Dai
CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End
  Code Review Evaluation in Python Projects
Automated code review (CR) is a key application for Large Language Models
(LLMs), but progress is hampered by a "reality gap": existing benchmarks
evaluate models on isolated sub-tasks using simplified, context-poor data. This
fails to reflect the holistic context-rich nature of real-world CR. To bridge
this gap, we introduce CodeFuse-CR-Bench, the first comprehensiveness-aware
benchmark for repository-level CR evaluation. CodeFuse-CR-Bench comprises 601
high-quality instances from 70 Python projects covering nine Pull-Request (PR)
problem domains, where each instance provides rich, multi-faceted context
including the associated issue, PR details, and repository state, enabling
end-to-end evaluation. Beyond superficial metrics, we also propose a novel
evaluation framework that combines rule-based checks for location and syntax
with model-based judgments of review quality. We present the first large-scale
assessment of state-of-the-art LLMs on this comprehensive CR task. Our results
establish crucial baselines and reveal that (1) no single LLM dominates all
aspects of CR; (2) Gemini 2.5 Pro achieves the highest comprehensive
performance; and (3) different LLMs exhibit varying robustness to redundant
context. These findings highlight the necessity of holistic, multi-dimensional
evaluation and provide actionable insights for advancing truly intelligent yet
practical CR assistants.
A new benchmark, CodeFuse-CR-Bench, evaluates LLMs in repository-level code review with comprehensive, context-rich data and a novel evaluation framework.
Large Language Models automated code review CodeFuse-CR-Bench repository-level CR evaluation Pull-Request problem domains rule-based checks model-based judgments review quality Gemini 2.5 Pro