CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research
Owen Queen, Harrison Zhang, James Zou
Variant and gene interpretation are vital prerequisites for clinical genetics and personalized medicine as they guide the diagnosis and management of rare or common diseases with strong genetic etiologies. However, traditional approaches for this task are manual and labor-intensive.Generative language models (LMs) provide an opportunity to facilitate the review of genes and variants, thereby accelerating the translation of genetic sequencing data into clinically-actionable insights. While existing benchmarks have attempted to quantify the capabilities of LMs for interpreting scientific data, these benchmarks often focus on narrow tasks that do not translate to real-world research.Harnessing the ClinGen database, a resource of expert-curated literature interpretations of publications in clinical genetics, we built CGBench, a robust benchmark that tests complex reasoning capabilities of LMs on scientific publications.The tasks in CGBench measure the ability to 1) extract relevant experimental results following precise protocols and guidelines,2) judge the strength of presented evidence, and 3) categorize and describe assays and experiments and their outcomes as is relevant to specific genes and variants.We test 8 different LMs on these tasks and find that while models show promise, substantial gaps still exist in abilities to correctly interpret literature, especially fine-grained instructions.Our experiments reveal that while reasoning models are best-performing across fine-grained instructions, non-reasoning models are still better at answering high-level questions about scientific data. Additionally, in-context learning of demonstrations can significantly boost performance.Our evaluation of explanations shows that models often hallucinate or misinterpret results even when correctly classifying evidence.CGBench introduces a rigorous, challenging benchmark that precisely measures scientific literature interpretation, revealing strengths and points of weakness for development of LMs and agentic systems.