Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and
  Reasoning
Scientific problem solving poses unique challenges for LLMs, requiring both
deep domain knowledge and the ability to apply such knowledge through complex
reasoning. While automated scientific reasoners hold great promise for
assisting human scientists, there is currently no widely adopted holistic
benchmark for evaluating scientific reasoning, and few approaches
systematically disentangle the distinct roles of knowledge and reasoning in
these tasks. To address these gaps, we introduce SciReas, a diverse suite of
existing benchmarks for scientific reasoning tasks, and SciReas-Pro, a
selective subset that requires more complex reasoning. Our holistic evaluation
surfaces insights about scientific reasoning performance that remain hidden
when relying on individual benchmarks alone. We then propose KRUX, a probing
framework for studying the distinct roles of reasoning and knowledge in
scientific tasks. Combining the two, we conduct an in-depth analysis that
yields several key findings: (1) Retrieving task-relevant knowledge from model
parameters is a critical bottleneck for LLMs in scientific reasoning; (2)
Reasoning models consistently benefit from external knowledge added in-context
on top of the reasoning enhancement; (3) Enhancing verbalized reasoning
improves LLMs' ability to surface task-relevant knowledge. Finally, we conduct
a lightweight analysis, comparing our science-focused data composition with
concurrent efforts on long CoT SFT, and release SciLit01, a strong 8B baseline
for scientific reasoning.
68aeac4f364411bea07df849
Alan Li
claimed_verified
2025-08-29T08:03:13.616Z
6209ab721c19916805e5b2d1
/avatars/67cef09d77ada966af323b477c321dd7.svg
Alan Li
user
lihaoxin2020
68aeac4f364411bea07df84a
Yixin Liu
68aeac4f364411bea07df84b
Arpan Sarkar
68aeac4f364411bea07df84c
Doug Downey
68aeac4f364411bea07df84d
Arman Cohan
Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and
  Reasoning
Scientific problem solving poses unique challenges for LLMs, requiring both
deep domain knowledge and the ability to apply such knowledge through complex
reasoning. While automated scientific reasoners hold great promise for
assisting human scientists, there is currently no widely adopted holistic
benchmark for evaluating scientific reasoning, and few approaches
systematically disentangle the distinct roles of knowledge and reasoning in
these tasks. To address these gaps, we introduce SciReas, a diverse suite of
existing benchmarks for scientific reasoning tasks, and SciReas-Pro, a
selective subset that requires more complex reasoning. Our holistic evaluation
surfaces insights about scientific reasoning performance that remain hidden
when relying on individual benchmarks alone. We then propose KRUX, a probing
framework for studying the distinct roles of reasoning and knowledge in
scientific tasks. Combining the two, we conduct an in-depth analysis that
yields several key findings: (1) Retrieving task-relevant knowledge from model
parameters is a critical bottleneck for LLMs in scientific reasoning; (2)
Reasoning models consistently benefit from external knowledge added in-context
on top of the reasoning enhancement; (3) Enhancing verbalized reasoning
improves LLMs' ability to surface task-relevant knowledge. Finally, we conduct
a lightweight analysis, comparing our science-focused data composition with
concurrent efforts on long CoT SFT, and release SciLit01, a strong 8B baseline
for scientific reasoning.
SciReas and SciReas-Pro benchmarks, along with KRUX framework, provide insights into the distinct roles of knowledge and reasoning in scientific tasks, highlighting critical bottlenecks and improvements for LLMs.
LLMs scientific reasoning SciReas SciReas-Pro KRUX task-relevant knowledge reasoning models external knowledge verbalized reasoning SciLit01