BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge
  Bases
Biomedical researchers increasingly rely on large-scale structured databases
for complex analytical tasks. However, current text-to-SQL systems often
struggle to map qualitative scientific questions into executable SQL,
particularly when implicit domain reasoning is required. We introduce
BiomedSQL, the first benchmark explicitly designed to evaluate scientific
reasoning in text-to-SQL generation over a real-world biomedical knowledge
base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in
a harmonized BigQuery knowledge base that integrates gene-disease associations,
causal inference from omics data, and drug approval records. Each question
requires models to infer domain-specific criteria, such as genome-wide
significance thresholds, effect directionality, or trial phase filtering,
rather than rely on syntactic translation alone. We evaluate a range of open-
and closed-source LLMs across prompting strategies and interaction paradigms.
Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0%
execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%,
both well below the expert baseline of 90.0%. BiomedSQL provides a new
foundation for advancing text-to-SQL systems capable of supporting scientific
discovery through robust reasoning over structured biomedical knowledge bases.
Our dataset is publicly available at
https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source
at https://github.com/NIH-CARD/biomedsql.
68370a124a68a691a29331cb
Mathew J. Koretsky
claimed_verified
2025-05-28T19:14:08.928Z
672e33ad6ff4cb430622f915
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_HBL0daPqqdN8rNU6l25i.png
Mathew Koretsky
user
mkoretsky1
68370a124a68a691a29331cc
Maya Willey
68370a124a68a691a29331cd
Adi Asija
68370a124a68a691a29331ce
Owen Bianchi
68370a124a68a691a29331cf
Chelsea X. Alvarado
68370a124a68a691a29331d0
Tanay Nayak
68370a124a68a691a29331d1
Nicole Kuznetsov
68370a124a68a691a29331d2
Sungwon Kim
68370a124a68a691a29331d3
Mike A. Nalls
68370a124a68a691a29331d4
Daniel Khashabi
claimed_verified
2025-05-28T19:14:06.658Z
5f6540c65e78cc6b0ed3199d
/avatars/0280d4df417855965a0964d22766c012.svg
Daniel Khashabi
user
danyaljj
68370a124a68a691a29331d5
Faraz Faghri
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge
  Bases
Biomedical researchers increasingly rely on large-scale structured databases
for complex analytical tasks. However, current text-to-SQL systems often
struggle to map qualitative scientific questions into executable SQL,
particularly when implicit domain reasoning is required. We introduce
BiomedSQL, the first benchmark explicitly designed to evaluate scientific
reasoning in text-to-SQL generation over a real-world biomedical knowledge
base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in
a harmonized BigQuery knowledge base that integrates gene-disease associations,
causal inference from omics data, and drug approval records. Each question
requires models to infer domain-specific criteria, such as genome-wide
significance thresholds, effect directionality, or trial phase filtering,
rather than rely on syntactic translation alone. We evaluate a range of open-
and closed-source LLMs across prompting strategies and interaction paradigms.
Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0%
execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%,
both well below the expert baseline of 90.0%. BiomedSQL provides a new
foundation for advancing text-to-SQL systems capable of supporting scientific
discovery through robust reasoning over structured biomedical knowledge bases.
Our dataset is publicly available at
https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source
at https://github.com/NIH-CARD/biomedsql.
BiomedSQL evaluates scientific reasoning in text-to-SQL tasks using a large biomedical knowledge base, highlighting performance gaps in existing models.
text-to-SQL biomedical knowledge base gene-disease associations causal inference omics data drug approval records genome-wide significance effect directionality trial phase filtering multi-step agent expert baseline