CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code
  Completion
Code completion models have made significant progress in recent years, yet
current popular evaluation datasets, such as HumanEval and MBPP, predominantly
focus on code completion tasks within a single file. This over-simplified
setting falls short of representing the real-world software development
scenario where repositories span multiple files with numerous cross-file
dependencies, and accessing and understanding cross-file context is often
required to complete the code correctly.
  To fill in this gap, we propose CrossCodeEval, a diverse and multilingual
code completion benchmark that necessitates an in-depth cross-file contextual
understanding to complete the code accurately. CrossCodeEval is built on a
diverse set of real-world, open-sourced, permissively-licensed repositories in
four popular programming languages: Python, Java, TypeScript, and C#. To create
examples that strictly require cross-file context for accurate completion, we
propose a straightforward yet efficient static-analysis-based approach to
pinpoint the use of cross-file context within the current file.
  Extensive experiments on state-of-the-art code language models like CodeGen
and StarCoder demonstrate that CrossCodeEval is extremely challenging when the
relevant cross-file context is absent, and we see clear improvements when
adding these context into the prompt. However, despite such improvements, the
pinnacle of performance remains notably unattained even with the
highest-performing model, indicating that CrossCodeEval is also capable of
assessing model's capability in leveraging extensive context to make better
code completion. Finally, we benchmarked various methods in retrieving
cross-file context, and show that CrossCodeEval can also be used to measure the
capability of code retrievers.
652f4fa0f75d9cc2a4d418ad
Yangruibo Ding
admin_assigned
2023-10-18T09:30:02.354Z
6458648072b60ae7a381044f
/avatars/f6e26c8663b30cf8c742e494929859ea.svg
Yangruibo Ding
user
yrbding
652f4fa0f75d9cc2a4d418ae
Zijian Wang
admin_assigned
2023-10-18T09:30:28.894Z
6465183934acce85aea2c3b3
/avatars/275a0811381afc980d339ba2aa1294c7.svg
Zijian Wang
user
zijwang
652f4fa0f75d9cc2a4d418af
Wasi Uddin Ahmad
admin_assigned
2023-10-18T09:30:36.866Z
6115506463c5df6b8d45fb65
https://cdn-avatars.huggingface.co/v1/production/uploads/1671159107866-6115506463c5df6b8d45fb65.jpeg
Wasi Uddin Ahmad
user
wasiahmad
652f4fa0f75d9cc2a4d418b0
Hantian Ding
admin_assigned
2023-10-18T09:31:02.495Z
637556dafd59a439194d819d
/avatars/a6241a3adf3bd3b1c3a6ba63933a4117.svg
Hantian Ding
user
dhantian
652f4fa0f75d9cc2a4d418b1
Ming Tan
652f4fa0f75d9cc2a4d418b2
Nihal Jain
652f4fa0f75d9cc2a4d418b3
Murali Krishna Ramanathan
652f4fa0f75d9cc2a4d418b4
Ramesh Nallapati
652f4fa0f75d9cc2a4d418b5
Parminder Bhatia
652f4fa0f75d9cc2a4d418b6
Dan Roth
652f4fa0f75d9cc2a4d418b7
Bing Xiang
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code
  Completion
Code completion models have made significant progress in recent years, yet
current popular evaluation datasets, such as HumanEval and MBPP, predominantly
focus on code completion tasks within a single file. This over-simplified
setting falls short of representing the real-world software development
scenario where repositories span multiple files with numerous cross-file
dependencies, and accessing and understanding cross-file context is often
required to complete the code correctly.
  To fill in this gap, we propose CrossCodeEval, a diverse and multilingual
code completion benchmark that necessitates an in-depth cross-file contextual
understanding to complete the code accurately. CrossCodeEval is built on a
diverse set of real-world, open-sourced, permissively-licensed repositories in
four popular programming languages: Python, Java, TypeScript, and C#. To create
examples that strictly require cross-file context for accurate completion, we
propose a straightforward yet efficient static-analysis-based approach to
pinpoint the use of cross-file context within the current file.
  Extensive experiments on state-of-the-art code language models like CodeGen
and StarCoder demonstrate that CrossCodeEval is extremely challenging when the
relevant cross-file context is absent, and we see clear improvements when
adding these context into the prompt. However, despite such improvements, the
pinnacle of performance remains notably unattained even with the
highest-performing model, indicating that CrossCodeEval is also capable of
assessing model's capability in leveraging extensive context to make better
code completion. Finally, we benchmarked various methods in retrieving
cross-file context, and show that CrossCodeEval can also be used to measure the
capability of code retrievers.
CrossCodeEval is a code completion benchmark that tests models' ability to understand and use cross-file context from real-world, multilingual repositories, demonstrating the need for improved context retrieval and model performance.
CrossCodeEval code completion benchmark cross-file contextual understanding real-world repositories multilingual static-analysis prompt code retrievers