Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
The task of issue resolving is to modify a codebase to generate a patch that
addresses a given issue. However, existing benchmarks, such as SWE-bench, focus
almost exclusively on Python, making them insufficient for evaluating Large
Language Models (LLMs) across diverse software ecosystems. To address this, we
introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench,
covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a
total of 1,632 high-quality instances, which were carefully annotated from
2,456 candidates by 68 expert annotators, ensuring that the benchmark can
provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we
evaluate a series of state-of-the-art models using three representative methods
(Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with
key empirical insights. In addition, we launch a Multi-SWE-RL open-source
community, aimed at building large-scale reinforcement learning (RL) training
datasets for issue-resolving tasks. As an initial contribution, we release a
set of 4,723 well-structured instances spanning seven programming languages,
laying a solid foundation for RL research in this domain. More importantly, we
open-source our entire data production pipeline, along with detailed tutorials,
encouraging the open-source community to continuously contribute and expand the
dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL
community as catalysts for advancing RL toward its full potential, bringing us
one step closer to the dawn of AGI.
67ef4d92c1e251f239495a13
Daoguang Zan
claimed_verified
2025-04-08T06:56:29.085Z
61527edf8b55dbdae72874fa
https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
Daoguang Zan
user
Daoguang
67ef4d92c1e251f239495a14
Zhirong Huang
67ef4d92c1e251f239495a15
Wei Liu
67ef4d92c1e251f239495a16
Hanwu Chen
67ef4d92c1e251f239495a17
Linhao Zhang
67ef4d92c1e251f239495a18
Shulin Xin
67ef4d92c1e251f239495a19
Lu Chen
67ef4d92c1e251f239495a1a
Qi Liu
67ef4d92c1e251f239495a1b
Xiaojian Zhong
67ef4d92c1e251f239495a1c
Aoyan Li
67ef4d92c1e251f239495a1d
Siyao Liu
67ef4d92c1e251f239495a1e
Yongsheng Xiao
67ef4d92c1e251f239495a1f
Liangqiang Chen
67ef4d92c1e251f239495a20
Yuyu Zhang
67ef4d92c1e251f239495a21
Jing Su
67ef4d92c1e251f239495a22
Tianyu Liu
67ef4d92c1e251f239495a23
Rui Long
67ef4d92c1e251f239495a24
Kai Shen
67ef4d92c1e251f239495a25
Liang Xiang
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
The task of issue resolving is to modify a codebase to generate a patch that
addresses a given issue. However, existing benchmarks, such as SWE-bench, focus
almost exclusively on Python, making them insufficient for evaluating Large
Language Models (LLMs) across diverse software ecosystems. To address this, we
introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench,
covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a
total of 1,632 high-quality instances, which were carefully annotated from
2,456 candidates by 68 expert annotators, ensuring that the benchmark can
provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we
evaluate a series of state-of-the-art models using three representative methods
(Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with
key empirical insights. In addition, we launch a Multi-SWE-RL open-source
community, aimed at building large-scale reinforcement learning (RL) training
datasets for issue-resolving tasks. As an initial contribution, we release a
set of 4,723 well-structured instances spanning seven programming languages,
laying a solid foundation for RL research in this domain. More importantly, we
open-source our entire data production pipeline, along with detailed tutorials,
encouraging the open-source community to continuously contribute and expand the
dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL
community as catalysts for advancing RL toward its full potential, bringing us
one step closer to the dawn of AGI.
A multilingual benchmark, Multi-SWE-bench, is introduced to evaluate Large Language Models across various programming languages and is complemented by a RL dataset, Multi-SWE-RL, to advance issue-resolving tasks.
Large Language Models (LLMs) multilingual issue-resolving benchmark Multi-SWE-bench Java TypeScript JavaScript Go Rust C C++ high-quality instances expert annotators state-of-the-art models Agentless SWE-agent OpenHands reinforcement learning (RL) Multi-SWE-RL open-source community data production pipeline detail tutorials AGI