DevEval: A Manually-Annotated Code Generation Benchmark Aligned with
  Real-World Code Repositories
How to evaluate the coding abilities of Large Language Models (LLMs) remains
an open question. We find that existing benchmarks are poorly aligned with
real-world code repositories and are insufficient to evaluate the coding
abilities of LLMs.
  To address the knowledge gap, we propose a new benchmark named DevEval, which
has three advances. (1) DevEval aligns with real-world repositories in multiple
dimensions, e.g., code distributions and dependency distributions. (2) DevEval
is annotated by 13 developers and contains comprehensive annotations (e.g.,
requirements, original repositories, reference code, and reference
dependencies). (3) DevEval comprises 1,874 testing samples from 117
repositories, covering 10 popular domains (e.g., Internet, Database). Based on
DevEval, we propose repository-level code generation and evaluate 8 popular
LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa).
Our experiments reveal these LLMs' coding abilities in real-world code
repositories. For example, in our experiments, the highest Pass@1 of
gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize
their shortcomings. We hope DevEval can facilitate the development of LLMs in
real code repositories. DevEval, prompts, and LLMs' predictions have been
released.
6659eb90edd7468b910e7299
Jia Li
6659eb90edd7468b910e729a
Ge Li
claimed_verified
2024-10-10T08:02:46.558Z
6530b96ef68c6e013d08b413
/avatars/ecc2874e612d0aed357d072427faf21a.svg
Ge Li
user
ligechina
6659eb90edd7468b910e729b
Yunfei Zhao
6659eb90edd7468b910e729c
Yongmin Li
6659eb90edd7468b910e729d
Huanyu Liu
6659eb90edd7468b910e729e
Hao Zhu
6659eb90edd7468b910e729f
Lecheng Wang
6659eb90edd7468b910e72a0
Kaibo Liu
6659eb90edd7468b910e72a1
Zheng Fang
6659eb90edd7468b910e72a2
Lanshen Wang
6659eb90edd7468b910e72a3
Jiazheng Ding
6659eb90edd7468b910e72a4
Xuanming Zhang
admin_assigned
2024-05-31T19:50:15.673Z
65fc5109899083a2aad987c5
/avatars/289dbb8128746d931118cff6f6871a45.svg
XUANMING ZHANG
user
XUANMINGZHANG
6659eb90edd7468b910e72a5
Yuqi Zhu
admin_assigned
2024-05-31T19:49:57.424Z
646d3c3aeb9268aeebc37de1
/avatars/877aff144e43f93b390cf05f95ddf11c.svg
Yuqi Zhu
user
YuqiZhu29
6659eb90edd7468b910e72a6
Yihong Dong
admin_assigned
2024-05-31T19:49:50.485Z
6468e631b2321e47d3277ec9
/avatars/3acd3868232067289fad69db6c32cf87.svg
Yihong Dong
user
dongyh
6659eb90edd7468b910e72a7
Zhi Jin
6659eb90edd7468b910e72a8
Binhua Li
admin_assigned
2024-05-31T19:49:43.159Z
656ddba4271c5c4e3356cf9d
/avatars/88e47148c2dd0dc171d103d2a0e9f429.svg
Binhua
user
BinhuaL
6659eb90edd7468b910e72a9
Fei Huang
admin_assigned
2024-05-31T19:49:12.851Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
6659eb90edd7468b910e72aa
Yongbin Li
claimed_verified
2024-10-14T10:48:05.019Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
user
Yongbin-Li
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with
  Real-World Code Repositories
How to evaluate the coding abilities of Large Language Models (LLMs) remains
an open question. We find that existing benchmarks are poorly aligned with
real-world code repositories and are insufficient to evaluate the coding
abilities of LLMs.
  To address the knowledge gap, we propose a new benchmark named DevEval, which
has three advances. (1) DevEval aligns with real-world repositories in multiple
dimensions, e.g., code distributions and dependency distributions. (2) DevEval
is annotated by 13 developers and contains comprehensive annotations (e.g.,
requirements, original repositories, reference code, and reference
dependencies). (3) DevEval comprises 1,874 testing samples from 117
repositories, covering 10 popular domains (e.g., Internet, Database). Based on
DevEval, we propose repository-level code generation and evaluate 8 popular
LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa).
Our experiments reveal these LLMs' coding abilities in real-world code
repositories. For example, in our experiments, the highest Pass@1 of
gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize
their shortcomings. We hope DevEval can facilitate the development of LLMs in
real code repositories. DevEval, prompts, and LLMs' predictions have been
released.
repository-level code generation