CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End
  Code Review Evaluation in Python Projects
Automated code review (CR) is a key application for Large Language Models
(LLMs), but progress is hampered by a "reality gap": existing benchmarks
evaluate models on isolated sub-tasks using simplified, context-poor data. This
fails to reflect the holistic context-rich nature of real-world CR. To bridge
this gap, we introduce CodeFuse-CR-Bench, the first comprehensiveness-aware
benchmark for repository-level CR evaluation. CodeFuse-CR-Bench comprises 601
high-quality instances from 70 Python projects covering nine Pull-Request (PR)
problem domains, where each instance provides rich, multi-faceted context
including the associated issue, PR details, and repository state, enabling
end-to-end evaluation. Beyond superficial metrics, we also propose a novel
evaluation framework that combines rule-based checks for location and syntax
with model-based judgments of review quality. We present the first large-scale
assessment of state-of-the-art LLMs on this comprehensive CR task. Our results
establish crucial baselines and reveal that (1) no single LLM dominates all
aspects of CR; (2) Gemini 2.5 Pro achieves the highest comprehensive
performance; and (3) different LLMs exhibit varying robustness to redundant
context. These findings highlight the necessity of holistic, multi-dimensional
evaluation and provide actionable insights for advancing truly intelligent yet
practical CR assistants.
68d2371c1ca7156988a8ee2e
Hanyang Guo
68d2371c1ca7156988a8ee2f
Xunjin Zheng
68d2371c1ca7156988a8ee30
Zihan Liao
68d2371c1ca7156988a8ee31
Hang Yu
68d2371c1ca7156988a8ee32
Peng DI
68d2371c1ca7156988a8ee33
Ziyin Zhang
claimed_verified
2025-09-23T10:05:07.145Z
6430bdd8cd31d174a9f900fb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
Ziyin Zhang
user
Geralt-Targaryen
68d2371c1ca7156988a8ee34
Hong-Ning Dai
CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End
  Code Review Evaluation in Python Projects
Automated code review (CR) is a key application for Large Language Models
(LLMs), but progress is hampered by a "reality gap": existing benchmarks
evaluate models on isolated sub-tasks using simplified, context-poor data. This
fails to reflect the holistic context-rich nature of real-world CR. To bridge
this gap, we introduce CodeFuse-CR-Bench, the first comprehensiveness-aware
benchmark for repository-level CR evaluation. CodeFuse-CR-Bench comprises 601
high-quality instances from 70 Python projects covering nine Pull-Request (PR)
problem domains, where each instance provides rich, multi-faceted context
including the associated issue, PR details, and repository state, enabling
end-to-end evaluation. Beyond superficial metrics, we also propose a novel
evaluation framework that combines rule-based checks for location and syntax
with model-based judgments of review quality. We present the first large-scale
assessment of state-of-the-art LLMs on this comprehensive CR task. Our results
establish crucial baselines and reveal that (1) no single LLM dominates all
aspects of CR; (2) Gemini 2.5 Pro achieves the highest comprehensive
performance; and (3) different LLMs exhibit varying robustness to redundant
context. These findings highlight the necessity of holistic, multi-dimensional
evaluation and provide actionable insights for advancing truly intelligent yet
practical CR assistants.
A new benchmark, CodeFuse-CR-Bench, evaluates LLMs in repository-level code review with comprehensive, context-rich data and a novel evaluation framework.
Large Language Models automated code review CodeFuse-CR-Bench repository-level CR evaluation Pull-Request problem domains rule-based checks model-based judgments review quality Gemini 2.5 Pro