PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation
Large language models (LLMS) have shown increasing effectiveness in
Text-to-SQL tasks. However, another closely related problem, Cross-System SQL
Translation (a.k.a., SQL-to-SQL), which adapts a query written for one database
system (e.g., MySQL) into its equivalent one for another system (e.g.,
ClickHouse), is of great practical importance but remains underexplored.
Existing SQL benchmarks are not well-suited for SQL-to-SQL evaluation, which
(1) focus on a limited set of database systems (often just SQLite) and (2)
cannot capture many system-specific SQL dialects (e.g., customized functions,
data types, and syntax rules). Thus, in this paper, we introduce PARROT, a
Practical And Realistic BenchmaRk for CrOss-System SQL Translation. PARROT
comprises 598 translation pairs from 38 open-source benchmarks and real-world
business services, specifically prepared to challenge system-specific SQL
understanding (e.g., LLMS achieve lower than 38.53% accuracy on average). We
also provide multiple benchmark variants, including PARROT-Diverse with 28,003
translations (for extensive syntax testing) and PARROT-Simple with 5,306
representative samples (for focused stress testing), covering 22
production-grade database systems. To promote future research, we release a
public leaderboard and source code at: https://code4db.github.io/parrot-bench/.
68db7c7dd2bf1f4b15ec778b
Wei Zhou
68db7c7dd2bf1f4b15ec778c
Guoliang Li
68db7c7dd2bf1f4b15ec778d
Haoyu Wang
68db7c7dd2bf1f4b15ec778e
Yuxing Han
68db7c7dd2bf1f4b15ec778f
Xufei Wu
68db7c7dd2bf1f4b15ec7790
Fan Wu
68db7c7dd2bf1f4b15ec7791
Xuanhe Zhou
PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation
Large language models (LLMS) have shown increasing effectiveness in
Text-to-SQL tasks. However, another closely related problem, Cross-System SQL
Translation (a.k.a., SQL-to-SQL), which adapts a query written for one database
system (e.g., MySQL) into its equivalent one for another system (e.g.,
ClickHouse), is of great practical importance but remains underexplored.
Existing SQL benchmarks are not well-suited for SQL-to-SQL evaluation, which
(1) focus on a limited set of database systems (often just SQLite) and (2)
cannot capture many system-specific SQL dialects (e.g., customized functions,
data types, and syntax rules). Thus, in this paper, we introduce PARROT, a
Practical And Realistic BenchmaRk for CrOss-System SQL Translation. PARROT
comprises 598 translation pairs from 38 open-source benchmarks and real-world
business services, specifically prepared to challenge system-specific SQL
understanding (e.g., LLMS achieve lower than 38.53% accuracy on average). We
also provide multiple benchmark variants, including PARROT-Diverse with 28,003
translations (for extensive syntax testing) and PARROT-Simple with 5,306
representative samples (for focused stress testing), covering 22
production-grade database systems. To promote future research, we release a
public leaderboard and source code at: https://code4db.github.io/parrot-bench/.
PARROT is a benchmark for evaluating Cross-System SQL Translation across multiple database systems, addressing limitations in existing SQL benchmarks.
Large language models Text-to-SQL Cross-System SQL Translation SQL-to-SQL database systems SQL dialects PARROT benchmark system-specific SQL public leaderboard