CodeEditorBench: Evaluating Code Editing Capability of Large Language
  Models
Large Language Models (LLMs) for code are rapidly evolving, with code editing
emerging as a critical capability. We introduce CodeEditorBench, an evaluation
framework designed to rigorously assess the performance of LLMs in code editing
tasks, including debugging, translating, polishing, and requirement switching.
Unlike existing benchmarks focusing solely on code generation, CodeEditorBench
emphasizes real-world scenarios and practical aspects of software development.
We curate diverse coding challenges and scenarios from five sources, covering
various programming languages, complexity levels, and editing tasks. Evaluation
of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and
GPT-4), outperform open-source models in CodeEditorBench, highlighting
differences in model performance based on problem types and prompt
sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by
providing a robust platform for assessing code editing capabilities. We will
release all prompts and datasets to enable the community to expand the dataset
and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to
the advancement of LLMs in code editing and provide a valuable resource for
researchers and practitioners.
660f55529bd915fe6b095cc2
Jiawei Guo
660f55529bd915fe6b095cc3
Ziming Li
660f55529bd915fe6b095cc4
Xueling Liu
660f55529bd915fe6b095cc5
Kaijing Ma
660f55529bd915fe6b095cc6
Tianyu Zheng
claimed_verified
2024-10-21T07:54:28.404Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
660f55529bd915fe6b095cc7
Zhouliang Yu
claimed_verified
2025-04-09T14:37:42.685Z
62a80fe3ac97233f1625235a
https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
Zhouliang Yu
user
zhouliang
660f55529bd915fe6b095cc8
Ding Pan
660f55529bd915fe6b095cc9
Yizhi LI
660f55529bd915fe6b095cca
Ruibo Liu
660f55529bd915fe6b095ccb
Yue Wang
660f55529bd915fe6b095ccc
Shuyue Guo
660f55529bd915fe6b095ccd
Xingwei Qu
660f55529bd915fe6b095cce
Xiang Yue
claimed_verified
2024-05-27T07:21:10.097Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
660f55529bd915fe6b095ccf
Ge Zhang
claimed_verified
2024-04-05T10:13:12.793Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
660f55529bd915fe6b095cd0
Wenhu Chen
660f55529bd915fe6b095cd1
Jie Fu
claimed_verified
2024-04-06T15:19:38.155Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
CodeEditorBench: Evaluating Code Editing Capability of Large Language
  Models
Large Language Models (LLMs) for code are rapidly evolving, with code editing
emerging as a critical capability. We introduce CodeEditorBench, an evaluation
framework designed to rigorously assess the performance of LLMs in code editing
tasks, including debugging, translating, polishing, and requirement switching.
Unlike existing benchmarks focusing solely on code generation, CodeEditorBench
emphasizes real-world scenarios and practical aspects of software development.
We curate diverse coding challenges and scenarios from five sources, covering
various programming languages, complexity levels, and editing tasks. Evaluation
of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and
GPT-4), outperform open-source models in CodeEditorBench, highlighting
differences in model performance based on problem types and prompt
sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by
providing a robust platform for assessing code editing capabilities. We will
release all prompts and datasets to enable the community to expand the dataset
and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to
the advancement of LLMs in code editing and provide a valuable resource for
researchers and practitioners.
CodeEditorBench evaluates Large Language Models in real-world code editing tasks, demonstrating that closed-source models perform better across various scenarios and problems.
Large Language Models LLMs code editing debugging translating polishing requirement switching code generation Gemini-Ultra GPT-4 code editing capabilities prompts datasets