REPOEXEC: Evaluate Code Generation with a Repository-Level Executable
  Benchmark
The ability of CodeLLMs to generate executable and functionally correct code
at the repository-level scale remains largely unexplored. We introduce
RepoExec, a novel benchmark for evaluating code generation at the
repository-level scale. RepoExec focuses on three main aspects: executability,
functional correctness through automated test case generation with high
coverage rate, and carefully crafted cross-file contexts to accurately generate
code. Our work explores a controlled scenario where developers specify
necessary code dependencies, challenging the model to integrate these
accurately. Experiments show that while pretrained LLMs outperform
instruction-tuned models in correctness, the latter excel in utilizing provided
dependencies and demonstrating debugging capabilities. We also introduce a new
instruction-tuned dataset that focuses on code dependencies and demonstrate
that CodeLLMs fine-tuned on our dataset have a better capability to leverage
these dependencies effectively. RepoExec aims to provide a comprehensive
evaluation of code functionality and alignment with developer intent, paving
the way for more reliable and applicable CodeLLMs in real-world scenarios. The
dataset and source code can be found
at~https://github.com/FSoft-AI4Code/RepoExec.
6674fc013ae9e9c65996d66e
Nam Le Hai
claimed_verified
2024-10-08T08:17:37.057Z
63a27e21f30c4642278de3ea
https://cdn-avatars.huggingface.co/v1/production/uploads/63a27e21f30c4642278de3ea/zj0mGDv9DEK9JYHFjfaZq.jpeg
Le Hai Nam
user
NamCyan
6674fc013ae9e9c65996d66f
Dung Manh Nguyen
admin_assigned
2024-06-21T11:48:53.306Z
6177978d1e3e9ecbf77ddeef
/avatars/607e28f09ee8995c9a13579ad46f25ec.svg
Dung Manh Nguyen
user
nmd2000
6674fc013ae9e9c65996d670
Nghi D. Q. Bui
admin_assigned
2024-06-21T11:49:46.436Z
63a44141de134926a2ba0458
https://cdn-avatars.huggingface.co/v1/production/uploads/63a44141de134926a2ba0458/FP3uxpsKL1nBrrKyiFne3.jpeg
Nghi Bui
user
bdqnghi
REPOEXEC: Evaluate Code Generation with a Repository-Level Executable
  Benchmark
The ability of CodeLLMs to generate executable and functionally correct code
at the repository-level scale remains largely unexplored. We introduce
RepoExec, a novel benchmark for evaluating code generation at the
repository-level scale. RepoExec focuses on three main aspects: executability,
functional correctness through automated test case generation with high
coverage rate, and carefully crafted cross-file contexts to accurately generate
code. Our work explores a controlled scenario where developers specify
necessary code dependencies, challenging the model to integrate these
accurately. Experiments show that while pretrained LLMs outperform
instruction-tuned models in correctness, the latter excel in utilizing provided
dependencies and demonstrating debugging capabilities. We also introduce a new
instruction-tuned dataset that focuses on code dependencies and demonstrate
that CodeLLMs fine-tuned on our dataset have a better capability to leverage
these dependencies effectively. RepoExec aims to provide a comprehensive
evaluation of code functionality and alignment with developer intent, paving
the way for more reliable and applicable CodeLLMs in real-world scenarios. The
dataset and source code can be found
at~https://github.com/FSoft-AI4Code/RepoExec.
RepoExec is a benchmark for evaluating repository-level code generation focusing on executability, functional correctness, and dependency integration.
LLMs CodeLLMs RepoExec executable functional correctness automated test case generation cross-file contexts code dependencies fine-tuned dataset debugging capabilities