Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'
Large language models (LLMs) have shown remarkable ability in code generation
with more than 90 pass@1 in solving Python coding problems in HumanEval and
MBPP. Such high accuracy leads to the question: can LLMs replace human
programmers? Existing manual crafted, simple, or single-line code generation
benchmarks cannot answer this question due to their gap with real-world
software development. To answer this question, we propose REPOCOD, a code
generation benchmark with 980 problems collected from 11 popular real-world
projects, with more than 58% of them requiring file-level or repository-level
context information. In addition, REPOCOD has the longest average canonical
solution length (331.6 tokens) and the highest average cyclomatic complexity
(9.00) compared to existing benchmarks. In our evaluations on ten LLMs, none of
the models can achieve more than 30 pass@1 on REPOCOD, disclosing the necessity
of building stronger LLMs that can help developers in real-world software
development.
672282a1125cc4e9e2715f76
Shanchao Liang
extracted_pending
2024-10-30T19:01:54.565Z
6604b3c18191f5a7efa86eba
/avatars/1ee0ae7e999f1fb3b068297eabb1bc8f.svg
Shanchao Liang
user
shanchao
672282a1125cc4e9e2715f77
Yiran Hu
extracted_confirmed
2024-10-30T21:09:13.540Z
658bb78f2e1f5bf05c2a3da9
https://cdn-avatars.huggingface.co/v1/production/uploads/658bb78f2e1f5bf05c2a3da9/c4keEyiRRGXww0kZQdQph.jpeg
Yiran Hu
user
Yiran-Hu1007
672282a1125cc4e9e2715f78
Nan Jiang
extracted_confirmed
2024-10-30T20:31:21.269Z
629e4ca2f2bda18349b6d330
https://cdn-avatars.huggingface.co/v1/production/uploads/629e4ca2f2bda18349b6d330/hd5XwxRQO32T12aWuK8lP.jpeg
Nan Jiang
user
jiang719
672282a1125cc4e9e2715f79
Lin Tan
claimed_verified
2024-10-31T09:54:53.523Z
65831342f9c5cda913df366a
https://cdn-avatars.huggingface.co/v1/production/uploads/65831342f9c5cda913df366a/h7MLYt--shRYQj4-q5XmR.jpeg
Lin Tan
user
lin-tan
Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'
Large language models (LLMs) have shown remarkable ability in code generation
with more than 90 pass@1 in solving Python coding problems in HumanEval and
MBPP. Such high accuracy leads to the question: can LLMs replace human
programmers? Existing manual crafted, simple, or single-line code generation
benchmarks cannot answer this question due to their gap with real-world
software development. To answer this question, we propose REPOCOD, a code
generation benchmark with 980 problems collected from 11 popular real-world
projects, with more than 58% of them requiring file-level or repository-level
context information. In addition, REPOCOD has the longest average canonical
solution length (331.6 tokens) and the highest average cyclomatic complexity
(9.00) compared to existing benchmarks. In our evaluations on ten LLMs, none of
the models can achieve more than 30 pass@1 on REPOCOD, disclosing the necessity
of building stronger LLMs that can help developers in real-world software
development.
REPOCOD, a new code generation benchmark using real-world software development problems, shows that existing LLMs do not perform well and highlights the need for more robust models to assist developers.
Large language models LLMS code generation HumanEval MBPP REPOCOD file-level context repository-level context cyclomatic complexity pass@1