SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering
  Tasks?
We introduce SWE-Bench Pro, a substantially more challenging benchmark that
builds upon the best practices of SWE-BENCH [25], but is explicitly designed to
capture realistic, complex, enterprise-level problems beyond the scope of
SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of
41 actively maintained repositories spanning business applications, B2B
services, and developer tools. The benchmark is partitioned into a public set
with open access to problems sourced from 11 repositories, a held-out set of 12
repositories and a commercial set of 18 proprietary repositories where we have
formal partnership agreements with early-stage startups. Problems in the
held-out and the commercial set are not publicly accessible, but we release
results on the commercial set. Our benchmark features long-horizon tasks that
may require hours to days for a professional software engineer to complete,
often involving patches across multiple files and substantial code
modifications. All tasks are human-verified and augmented with sufficient
context to ensure resolvability. In our evaluation of widely used coding
models, under a unified scaffold, we observe that their performance on
SWE-Bench PRO remains below 25% (Pass@1), with GPT-5 achieving the highest
score to date at 23.3%. To better understand these limitations, we cluster the
failure modes observed in the collected agent trajectories for a clearer
characterization of the error patterns exhibited by current models. Overall,
SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully
captures the complexity and diversity of real-world software development,
advancing the pursuit of truly autonomous software engineering agents at a
professional level.
68d2125b1ca7156988a8ed84
Xiang Deng
68d2125b1ca7156988a8ed85
Jeff Da
claimed_verified
2025-09-24T13:55:20.117Z
5fcacfec6d942eaf4d1ebba6
/avatars/cccd2fc038868005d089f9cab9ec462d.svg
Jeff Da
user
jda
68d2125b1ca7156988a8ed86
Edwin Pan
68d2125b1ca7156988a8ed87
Yannis Yiming He
68d2125b1ca7156988a8ed88
Charles Ide
68d2125b1ca7156988a8ed89
Kanak Garg
68d2125b1ca7156988a8ed8a
Niklas Lauffer
admin_assigned
2025-09-23T10:13:43.269Z
64cc2820ec160b67ca975aef
/avatars/7f8c52bed96a08eb59c8f76fc6988afb.svg
Niklas Lauffer
user
nlauffer
68d2125b1ca7156988a8ed8b
Andrew Park
68d2125b1ca7156988a8ed8c
Nitin Pasari
68d2125b1ca7156988a8ed8d
Chetan Rane
68d2125b1ca7156988a8ed8e
Karmini Sampath
68d2125b1ca7156988a8ed8f
Maya Krishnan
admin_assigned
2025-09-23T14:13:50.713Z
6848eb0671d4a5a3d5bbfe89
/avatars/b1965ea0151318c51b385c9fed473d77.svg
Maya Krishnan
user
MayaKrishnan
68d2125b1ca7156988a8ed90
Srivatsa Kundurthy
admin_assigned
2025-09-23T14:14:00.973Z
6303d604eedc089484c59c43
/avatars/cca9a85dcffe61f1489b5acde2294536.svg
Srivatsa Kundurthy
user
srivatsa-kundurthy
68d2125b1ca7156988a8ed91
Sean Hendryx
68d2125b1ca7156988a8ed92
Zifan Wang
68d2125b1ca7156988a8ed93
Chen Bo Calvin Zhang
admin_assigned
2025-09-23T14:13:37.677Z
63c47e9c8d95a5c7706a3249
/avatars/68a2605cc5608dd61e991310fba3b4c0.svg
Chen Bo Calvin Zhang
user
calvincbzhang
68d2125b1ca7156988a8ed94
Noah Jacobson
68d2125b1ca7156988a8ed95
Bing Liu
68d2125b1ca7156988a8ed96
Brad Kenstler
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering
  Tasks?
We introduce SWE-Bench Pro, a substantially more challenging benchmark that
builds upon the best practices of SWE-BENCH [25], but is explicitly designed to
capture realistic, complex, enterprise-level problems beyond the scope of
SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of
41 actively maintained repositories spanning business applications, B2B
services, and developer tools. The benchmark is partitioned into a public set
with open access to problems sourced from 11 repositories, a held-out set of 12
repositories and a commercial set of 18 proprietary repositories where we have
formal partnership agreements with early-stage startups. Problems in the
held-out and the commercial set are not publicly accessible, but we release
results on the commercial set. Our benchmark features long-horizon tasks that
may require hours to days for a professional software engineer to complete,
often involving patches across multiple files and substantial code
modifications. All tasks are human-verified and augmented with sufficient
context to ensure resolvability. In our evaluation of widely used coding
models, under a unified scaffold, we observe that their performance on
SWE-Bench PRO remains below 25% (Pass@1), with GPT-5 achieving the highest
score to date at 23.3%. To better understand these limitations, we cluster the
failure modes observed in the collected agent trajectories for a clearer
characterization of the error patterns exhibited by current models. Overall,
SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully
captures the complexity and diversity of real-world software development,
advancing the pursuit of truly autonomous software engineering agents at a
professional level.
SWE-Bench Pro is a challenging benchmark for coding models, featuring complex, enterprise-level problems that require substantial code modifications, with performance evaluations showing significant limitations in current models.