SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
GitHub issue resolving is a critical task in software engineering, recently
gaining significant attention in both industry and academia. Within this task,
SWE-bench has been released to evaluate issue resolving capabilities of large
language models (LLMs), but has so far only focused on Python version. However,
supporting more programming languages is also important, as there is a strong
demand in industry. As a first step toward multilingual support, we have
developed a Java version of SWE-bench, called SWE-bench-java. We have publicly
released the dataset, along with the corresponding Docker-based evaluation
environment and leaderboard, which will be continuously maintained and updated
in the coming months. To verify the reliability of SWE-bench-java, we implement
a classic method SWE-agent and test several powerful LLMs on it. As is well
known, developing a high-quality multi-lingual benchmark is time-consuming and
labor-intensive, so we welcome contributions through pull requests or
collaboration to accelerate its iteration and refinement, paving the way for
fully automated programming.
66cd3ae3a7af45d8de3a6343
Daoguang Zan
claimed_verified
2024-08-27T07:38:29.425Z
61527edf8b55dbdae72874fa
https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
Daoguang Zan
user
Daoguang
66cd3ae3a7af45d8de3a6344
Zhirong Huang
admin_assigned
2024-08-27T10:21:37.128Z
658d32dfbe1495eb89635bfc
/avatars/9493cfade840616dad627140dcb9e383.svg
huang zhirong
user
tonymgg
66cd3ae3a7af45d8de3a6345
Ailun Yu
admin_assigned
2024-08-27T10:21:43.217Z
61b997ae22e5b0fdd501a0ec
https://cdn-avatars.huggingface.co/v1/production/uploads/1639552935869-noauth.png
Ailun Yu
user
Tannpopo
66cd3ae3a7af45d8de3a6346
Shaoxin Lin
admin_assigned
2024-08-27T10:21:54.369Z
64c8e32cbaa60107a6a543be
/avatars/638dc910e4a4b525bfdadc601087a9c6.svg
linshaoxin
user
linshaoxin
66cd3ae3a7af45d8de3a6347
Yifan Shi
admin_assigned
2024-08-27T10:21:59.830Z
66abada7c82d0b52b2a6b655
/avatars/095f0e5e78bfb8c7fe0911b13b81ecce.svg
Yifan Shi
user
Yifannn
66cd3ae3a7af45d8de3a6348
Wei Liu
66cd3ae3a7af45d8de3a6349
Dong Chen
66cd3ae3a7af45d8de3a634a
Zongshuai Qi
admin_assigned
2024-08-27T10:22:30.764Z
66cd4056332ce5c7468455da
/avatars/bec2a903fcdf9299261cb50f5c978eb7.svg
Zongshuai Qi
user
kylin818
66cd3ae3a7af45d8de3a634b
Hao Yu
66cd3ae3a7af45d8de3a634c
Lei Yu
claimed_verified
2024-08-29T07:23:37.502Z
634786226f8773f2a28dfa5e
/avatars/cdf5352e1346b1b61f2de9ec98abc96e.svg
yulei
user
yulei123aaa
66cd3ae3a7af45d8de3a634d
Dezhi Ran
admin_assigned
2024-08-27T10:23:08.635Z
63ba68ec0a9866b28cb2d53e
/avatars/11cdb3f2cac4752cf430a5c98b0ae3a4.svg
Dezhi Ran
user
DezhiRan
66cd3ae3a7af45d8de3a634e
Muhan Zeng
66cd3ae3a7af45d8de3a634f
Bo Shen
66cd3ae3a7af45d8de3a6350
Pan Bian
66cd3ae3a7af45d8de3a6351
Guangtai Liang
66cd3ae3a7af45d8de3a6352
Bei Guan
66cd3ae3a7af45d8de3a6353
Pengjie Huang
66cd3ae3a7af45d8de3a6354
Tao Xie
66cd3ae3a7af45d8de3a6355
Yongji Wang
admin_assigned
2024-08-27T10:42:51.683Z
645c3d8a1d2ba8a1cf48683a
/avatars/152eb17d5918edc051804b4cbc508c38.svg
Wang
user
Yongji
66cd3ae3a7af45d8de3a6356
Qianxiang Wang
admin_assigned
2024-08-27T10:42:33.977Z
632db5eba0fd0d71dcc17290
/avatars/6c7b096d495d1128e19d4fb18789e129.svg
Qianxiang Wang
user
WangQ
SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
GitHub issue resolving is a critical task in software engineering, recently
gaining significant attention in both industry and academia. Within this task,
SWE-bench has been released to evaluate issue resolving capabilities of large
language models (LLMs), but has so far only focused on Python version. However,
supporting more programming languages is also important, as there is a strong
demand in industry. As a first step toward multilingual support, we have
developed a Java version of SWE-bench, called SWE-bench-java. We have publicly
released the dataset, along with the corresponding Docker-based evaluation
environment and leaderboard, which will be continuously maintained and updated
in the coming months. To verify the reliability of SWE-bench-java, we implement
a classic method SWE-agent and test several powerful LLMs on it. As is well
known, developing a high-quality multi-lingual benchmark is time-consuming and
labor-intensive, so we welcome contributions through pull requests or
collaboration to accelerate its iteration and refinement, paving the way for
fully automated programming.
A Java version of SWE-bench, SWE-bench-java, has been developed and released to evaluate the issue resolution capabilities of large language models in Java, along with a Docker-based environment and leaderboard.