LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL
Converting natural language questions into SQL queries (Text-to-SQL) enables
non-expert users to interact with relational databases and has long been a
central task for natural language interfaces to data. While the WikiSQL dataset
played a key role in early NL2SQL research, its usage has declined due to
structural and annotation issues, including case sensitivity inconsistencies,
data type mismatches, syntax errors, and unanswered questions. We present
LLMSQL, a systematic revision and transformation of WikiSQL designed for the
LLM era. We classify these errors and implement automated methods for cleaning
and re-annotation. To assess the impact of these improvements, we evaluated
multiple large language models (LLMs), including Gemma 3, LLaMA 3.2, Mistral
7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek R1 and
others. Rather than serving as an update, LLMSQL is introduced as an LLM-ready
benchmark: unlike the original WikiSQL, tailored for pointer-network models
selecting tokens from input, LLMSQL provides clean natural language questions
and full SQL queries as plain text, enabling straightforward generation and
evaluation for modern natural language-to-SQL models.
68e3984373e20ab577842182
Dzmitry Pihulski
claimed_verified
2025-10-06T12:48:15.616Z
6622019a30a5b0bceaea6bbd
https://cdn-avatars.huggingface.co/v1/production/uploads/6622019a30a5b0bceaea6bbd/RwjSxndkR08cZDzM-pq4n.jpeg
Dzmitry Pihulski
user
pihull
68e3984373e20ab577842183
Karol Charchut
68e3984373e20ab577842184
Viktoria Novogrodskaia
68e3984373e20ab577842185
Jan Kocoń
LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL
Converting natural language questions into SQL queries (Text-to-SQL) enables
non-expert users to interact with relational databases and has long been a
central task for natural language interfaces to data. While the WikiSQL dataset
played a key role in early NL2SQL research, its usage has declined due to
structural and annotation issues, including case sensitivity inconsistencies,
data type mismatches, syntax errors, and unanswered questions. We present
LLMSQL, a systematic revision and transformation of WikiSQL designed for the
LLM era. We classify these errors and implement automated methods for cleaning
and re-annotation. To assess the impact of these improvements, we evaluated
multiple large language models (LLMs), including Gemma 3, LLaMA 3.2, Mistral
7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek R1 and
others. Rather than serving as an update, LLMSQL is introduced as an LLM-ready
benchmark: unlike the original WikiSQL, tailored for pointer-network models
selecting tokens from input, LLMSQL provides clean natural language questions
and full SQL queries as plain text, enabling straightforward generation and
evaluation for modern natural language-to-SQL models.
LLMSQL is a revised and cleaned version of WikiSQL designed for modern large language models, providing clean questions and full SQL queries for straightforward evaluation in text-to-SQL tasks.
Text-to-SQL NL2SQL WikiSQL LLM Gemma 3 LLaMA 3.2 Mistral 7B gpt-oss 20B Phi-3.5 Mini Qwen 2.5 OpenAI o4-mini DeepSeek R1 pointer-network models