SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in
  Real-World Applications
Resolution of complex SQL issues persists as a significant bottleneck in
real-world database applications. Current Large Language Models (LLMs), while
adept at text-to-SQL translation, have not been rigorously evaluated on the
more challenging task of debugging SQL issues. To address this gap, we
introduce BIRD-CRITIC, a new SQL issue debugging benchmark comprising 530
PostgreSQL tasks (BIRD-CRITIC-PG) and 570 multi-dialect tasks
(BIRD-CRITIC-Multi), distilled from authentic user issues and replayed within
new environments to facilitate rigorous evaluation. Baseline evaluations
underscore the task's complexity, with the leading reasoning model O3-Mini
achieving only 38.87% success rate on BIRD-CRITIC-PG and 33.33% on
BIRD-CRITIC-Multi. Meanwhile, advancing open-source models for database tasks
is crucial for empowering local development while safeguarding data privacy.
Therefore, we present Six-Gym (Sql-fIX-Gym), a training environment for
elevating open-source model capabilities for SQL issue debugging. This
environment leverages SQL-Rewind strategy, which automatically generates
executable issue-solution datasets by reverse-engineering issues from verified
SQLs. However, popular trajectory-based fine-tuning methods do not explore
substantial supervisory signals. We further propose f-Plan Boosting, which
extracts high-level debugging plans from SQL solutions, enabling teacher LLMs
to produce 73.7% more successful trajectories for training. We integrate these
components into an open-source agent, Bird-Fixer. Based on Qwen-2.5-Coder-14B,
Bird-Fixer achieves 38.11% success rate on BIRD-CRITIC-PG and 29.65% on
BIRD-CRITIC-Multi, surpassing leading proprietary models such as
Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing
sophisticated SQL-debugging capabilities. The leaderboard and source code are
available: https://bird-critic.github.io/
685ba757d2ee4fac76521f47
Jinyang Li
685ba757d2ee4fac76521f48
Xiaolong Li
claimed_verified
2025-06-25T08:07:05.354Z
653693cb8ee17cfd44eed8ce
/avatars/82be2428bec4e06c0a15a27647b9b8aa.svg
Xiaolong Li
user
xia01ongLi
685ba757d2ee4fac76521f49
Ge Qu
685ba757d2ee4fac76521f4a
Per Jacobsson
685ba757d2ee4fac76521f4b
Bowen Qin
685ba757d2ee4fac76521f4c
Binyuan Hui
685ba757d2ee4fac76521f4d
Shuzheng Si
claimed_verified
2025-06-26T09:32:40.580Z
637c99bbfe115289cfedfb44
https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
ssz
user
ssz1111
685ba757d2ee4fac76521f4e
Nan Huo
685ba757d2ee4fac76521f4f
Xiaohan Xu
claimed_verified
2025-06-25T09:19:04.596Z
63a3eb8af460e4379b5991e7
/avatars/7564a048d8496cac38d689178d90a8f9.svg
Xiaohan Xu
user
Tebmer
685ba757d2ee4fac76521f50
Yue Zhang
685ba757d2ee4fac76521f51
Ziwei Tang
685ba757d2ee4fac76521f52
Yuanshuai Li
685ba757d2ee4fac76521f53
Florensia Widjaja
claimed_verified
2025-06-25T20:59:20.731Z
665e7d0fddb96a4d725341ac
/avatars/30b235b10ee1436eb86c6a2ac884c904.svg
Florensia Widjaja
user
florensia712
685ba757d2ee4fac76521f54
Xintong Zhu
685ba757d2ee4fac76521f55
Feige Zhou
685ba757d2ee4fac76521f56
Yongfeng Huang
685ba757d2ee4fac76521f57
Yannis Papakonstantinou
685ba757d2ee4fac76521f58
Fatma Ozcan
685ba757d2ee4fac76521f59
Chenhao Ma
685ba757d2ee4fac76521f5a
Reynold Cheng
SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in
  Real-World Applications
Resolution of complex SQL issues persists as a significant bottleneck in
real-world database applications. Current Large Language Models (LLMs), while
adept at text-to-SQL translation, have not been rigorously evaluated on the
more challenging task of debugging SQL issues. To address this gap, we
introduce BIRD-CRITIC, a new SQL issue debugging benchmark comprising 530
PostgreSQL tasks (BIRD-CRITIC-PG) and 570 multi-dialect tasks
(BIRD-CRITIC-Multi), distilled from authentic user issues and replayed within
new environments to facilitate rigorous evaluation. Baseline evaluations
underscore the task's complexity, with the leading reasoning model O3-Mini
achieving only 38.87% success rate on BIRD-CRITIC-PG and 33.33% on
BIRD-CRITIC-Multi. Meanwhile, advancing open-source models for database tasks
is crucial for empowering local development while safeguarding data privacy.
Therefore, we present Six-Gym (Sql-fIX-Gym), a training environment for
elevating open-source model capabilities for SQL issue debugging. This
environment leverages SQL-Rewind strategy, which automatically generates
executable issue-solution datasets by reverse-engineering issues from verified
SQLs. However, popular trajectory-based fine-tuning methods do not explore
substantial supervisory signals. We further propose f-Plan Boosting, which
extracts high-level debugging plans from SQL solutions, enabling teacher LLMs
to produce 73.7% more successful trajectories for training. We integrate these
components into an open-source agent, Bird-Fixer. Based on Qwen-2.5-Coder-14B,
Bird-Fixer achieves 38.11% success rate on BIRD-CRITIC-PG and 29.65% on
BIRD-CRITIC-Multi, surpassing leading proprietary models such as
Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing
sophisticated SQL-debugging capabilities. The leaderboard and source code are
available: https://bird-critic.github.io/
A new benchmark and training environment for debugging SQL issues using advanced open-source models significantly improves their performance compared to proprietary solutions.
BIRD-CRITIC BIRD-CRITIC-PG BIRD-CRITIC-Multi PostgreSQL Six-Gym (Sql-fIX-Gym) SQL-Rewind f-Plan Boosting Bird-Fixer Qwen-2.5-Coder-14B Claude-3.7-Sonnet GPT-4.1