BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language
  Models via Lens of Dynamic Interactions
Large language models (LLMs) have demonstrated remarkable performance on
single-turn text-to-SQL tasks, but real-world database applications
predominantly require multi-turn interactions to handle ambiguous queries,
execution errors, and evolving user requirements. Existing multi-turn
benchmarks fall short by treating conversation histories as static context or
limiting evaluation to read-only operations, failing to reflect
production-grade database assistant challenges. We introduce BIRD-INTERACT, a
benchmark that restores this realism through: (1) a comprehensive interaction
environment coupling each database with a hierarchical knowledge base, metadata
files, and a function-driven user simulator, enabling models to solicit
clarifications, retrieve knowledge, and recover from errors without human
supervision; (2) two evaluation settings consisting of a pre-defined
conversational protocol (c-Interact) and an open-ended agentic setting
(a-Interact) where models autonomously decide when to query the user simulator
or explore the environment; (3) a challenging task suite covering the full CRUD
spectrum for business-intelligence and operational use cases, guarded by
executable test cases. Each task features ambiguous and follow-up sub-tasks
requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600
tasks, up to 11,796 interactions) for comprehensive performance assessment, and
BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed
behavioral analysis and rapid method development. Our empirical results
highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in
c-Interact and 17.00% in a-Interact. Analysis via memory grafting and
Interaction Test-time Scaling validates the importance of effective interaction
for complex, dynamic text-to-SQL tasks.
68e62536975ac4c405ef21d1
Nan Huo
claimed_verified
2025-10-15T15:25:39.567Z
60adfff0306d6873ec42d545
/avatars/4a63f90638dbffebfeeee181a6d0220c.svg
Nan
user
NanHUO
68e62536975ac4c405ef21d2
Xiaohan Xu
claimed_verified
2025-10-09T06:51:37.823Z
63a3eb8af460e4379b5991e7
/avatars/7564a048d8496cac38d689178d90a8f9.svg
Xiaohan Xu
user
Tebmer
68e62536975ac4c405ef21d3
Jinyang Li
68e62536975ac4c405ef21d4
Per Jacobsson
68e62536975ac4c405ef21d5
Shipei Lin
68e62536975ac4c405ef21d6
Bowen Qin
68e62536975ac4c405ef21d7
Binyuan Hui
68e62536975ac4c405ef21d8
Xiaolong Li
claimed_verified
2025-10-09T07:57:58.186Z
653693cb8ee17cfd44eed8ce
/avatars/82be2428bec4e06c0a15a27647b9b8aa.svg
Xiaolong Li
user
xia01ongLi
68e62536975ac4c405ef21d9
Ge Qu
68e62536975ac4c405ef21da
Shuzheng Si
68e62536975ac4c405ef21db
Linheng Han
68e62536975ac4c405ef21dc
Edward Alexander
68e62536975ac4c405ef21dd
Xintong Zhu
68e62536975ac4c405ef21de
Rui Qin
68e62536975ac4c405ef21df
Ruihan Yu
68e62536975ac4c405ef21e0
Yiyao Jin
68e62536975ac4c405ef21e1
Feige Zhou
68e62536975ac4c405ef21e2
Weihao Zhong
68e62536975ac4c405ef21e3
Yun Chen
68e62536975ac4c405ef21e4
Hongyu Liu
68e62536975ac4c405ef21e5
Chenhao Ma
68e62536975ac4c405ef21e6
Fatma Ozcan
68e62536975ac4c405ef21e7
Yannis Papakonstantinou
68e62536975ac4c405ef21e8
Reynold Cheng
BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language
  Models via Lens of Dynamic Interactions
Large language models (LLMs) have demonstrated remarkable performance on
single-turn text-to-SQL tasks, but real-world database applications
predominantly require multi-turn interactions to handle ambiguous queries,
execution errors, and evolving user requirements. Existing multi-turn
benchmarks fall short by treating conversation histories as static context or
limiting evaluation to read-only operations, failing to reflect
production-grade database assistant challenges. We introduce BIRD-INTERACT, a
benchmark that restores this realism through: (1) a comprehensive interaction
environment coupling each database with a hierarchical knowledge base, metadata
files, and a function-driven user simulator, enabling models to solicit
clarifications, retrieve knowledge, and recover from errors without human
supervision; (2) two evaluation settings consisting of a pre-defined
conversational protocol (c-Interact) and an open-ended agentic setting
(a-Interact) where models autonomously decide when to query the user simulator
or explore the environment; (3) a challenging task suite covering the full CRUD
spectrum for business-intelligence and operational use cases, guarded by
executable test cases. Each task features ambiguous and follow-up sub-tasks
requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600
tasks, up to 11,796 interactions) for comprehensive performance assessment, and
BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed
behavioral analysis and rapid method development. Our empirical results
highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in
c-Interact and 17.00% in a-Interact. Analysis via memory grafting and
Interaction Test-time Scaling validates the importance of effective interaction
for complex, dynamic text-to-SQL tasks.
BIRD-INTERACT is a benchmark for multi-turn text-to-SQL tasks that simulates realistic database assistant challenges through dynamic interactions, hierarchical knowledge bases, and autonomous decision-making.
large language models text-to-SQL multi-turn interactions conversation histories hierarchical knowledge base metadata files user simulator CRUD business-intelligence operational use cases memory grafting Interaction Test-time Scaling