SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
The issue-resolving task, where a model proposes patches to fix real-world bugs, has become a key benchmark for evaluating the capabilities of large language models (LLMs). SWE-bench is the prevailing benchmark in this domain, but it suffers from several limitations: it has not been updated since its release, is limited to just 12 repositories, and relies heavily on manual effort for both constructing test instances and setting up runnable environments, making it difficult to scale. We present SWE-bench-Live, a live-updatable benchmark designed to address these limitations. Our initial release includes 1,319 tasks derived from real GitHub issues created since 2024, spanning 93 repositories. Each task is accompanied by a dedicated Docker image to ensure reproducible execution. We introduce an automated curation pipeline that streamlines the entire process from instance creation to environment setup, removing manual bottlenecks and enabling continuous, scalable updates. We evaluate a range of state-of-the-art models and agent frameworks on SWE-bench-Live, offering detailed empirical insights into their real-world bug-fixing capabilities. By providing a fresh, diverse, and executable benchmark grounded in live repository activity, SWE-bench-Live supports reliable, large-scale assessment of code LLMs and code agents in realistic development settings.