From Medprompt to o1: Exploration of Run-Time Strategies for Medical
  Challenge Problems and Beyond
Run-time steering strategies like Medprompt are valuable for guiding large
language models (LLMs) to top performance on challenging tasks. Medprompt
demonstrates that a general LLM can be focused to deliver state-of-the-art
performance on specialized domains like medicine by using a prompt to elicit a
run-time strategy involving chain of thought reasoning and ensembling. OpenAI's
o1-preview model represents a new paradigm, where a model is designed to do
run-time reasoning before generating final responses. We seek to understand the
behavior of o1-preview on a diverse set of medical challenge problem
benchmarks. Following on the Medprompt study with GPT-4, we systematically
evaluate the o1-preview model across various medical benchmarks. Notably, even
without prompting techniques, o1-preview largely outperforms the GPT-4 series
with Medprompt. We further systematically study the efficacy of classic prompt
engineering strategies, as represented by Medprompt, within the new paradigm of
reasoning models. We found that few-shot prompting hinders o1's performance,
suggesting that in-context learning may no longer be an effective steering
approach for reasoning-native models. While ensembling remains viable, it is
resource-intensive and requires careful cost-performance optimization. Our cost
and accuracy analysis across run-time strategies reveals a Pareto frontier,
with GPT-4o representing a more affordable option and o1-preview achieving
state-of-the-art performance at higher cost. Although o1-preview offers top
performance, GPT-4o with steering strategies like Medprompt retains value in
specific contexts. Moreover, we note that the o1-preview model has reached
near-saturation on many existing medical benchmarks, underscoring the need for
new, challenging benchmarks. We close with reflections on general directions
for inference-time computation with LLMs.
672cedbe75014b062734d2ff
Harsha Nori
extracted_pending
2024-11-07T16:41:35.821Z
63336c425a032dcd095ba3fe
/avatars/72ce38aa86b4ada769539f0d91ce6937.svg
Harsha Nori
user
hanori
672cedbe75014b062734d300
Naoto Usuyama
claimed_verified
2024-11-15T09:32:49.980Z
5f243561e923d665e616263f
https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
Naoto Usuyama
user
naotous
672cedbe75014b062734d301
Nicholas King
672cedbe75014b062734d302
Scott Mayer McKinney
672cedbe75014b062734d303
Xavier Fernandes
672cedbe75014b062734d304
Sheng Zhang
claimed_verified
2025-05-09T07:23:39.770Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
672cedbe75014b062734d305
Eric Horvitz
From Medprompt to o1: Exploration of Run-Time Strategies for Medical
  Challenge Problems and Beyond
Run-time steering strategies like Medprompt are valuable for guiding large
language models (LLMs) to top performance on challenging tasks. Medprompt
demonstrates that a general LLM can be focused to deliver state-of-the-art
performance on specialized domains like medicine by using a prompt to elicit a
run-time strategy involving chain of thought reasoning and ensembling. OpenAI's
o1-preview model represents a new paradigm, where a model is designed to do
run-time reasoning before generating final responses. We seek to understand the
behavior of o1-preview on a diverse set of medical challenge problem
benchmarks. Following on the Medprompt study with GPT-4, we systematically
evaluate the o1-preview model across various medical benchmarks. Notably, even
without prompting techniques, o1-preview largely outperforms the GPT-4 series
with Medprompt. We further systematically study the efficacy of classic prompt
engineering strategies, as represented by Medprompt, within the new paradigm of
reasoning models. We found that few-shot prompting hinders o1's performance,
suggesting that in-context learning may no longer be an effective steering
approach for reasoning-native models. While ensembling remains viable, it is
resource-intensive and requires careful cost-performance optimization. Our cost
and accuracy analysis across run-time strategies reveals a Pareto frontier,
with GPT-4o representing a more affordable option and o1-preview achieving
state-of-the-art performance at higher cost. Although o1-preview offers top
performance, GPT-4o with steering strategies like Medprompt retains value in
specific contexts. Moreover, we note that the o1-preview model has reached
near-saturation on many existing medical benchmarks, underscoring the need for
new, challenging benchmarks. We close with reflections on general directions
for inference-time computation with LLMs.
o1-preview outperforms GPT-4 with Medprompt in medical benchmarks without prompting, highlighting the shift towards reasoning-native models and the need for new challenging benchmarks.
Medprompt large language models LLMs run-time reasoning chain of thought reasoning ensembling o1-preview in-context learning Pareto frontier GPT-4o