From Harm to Help: Turning Reasoning In-Context Demos into Assets for
  Reasoning LMs
Recent reasoning LLMs (RLMs), especially those trained with verifier-based
reinforcement learning, often perform worse with few-shot CoT than with direct
answering. We revisit this paradox using high-quality reasoning traces from
DeepSeek-R1 as demonstrations and find that adding more exemplars consistently
degrades accuracy, even when demonstrations are optimal. A detailed analysis
reveals two mechanisms behind this decline: (i) semantic misguidance, where
high textual similarity leads the model to treat the target as the same as the
exemplar and to copy intermediate steps verbatim; and (ii) strategy transfer
failure, where the model struggles to extract useful reasoning strategies and
apply them to target questions. Guided by these, we introduce Insight-to-Solve
(I2S), a sequential test-time procedure that turns demonstrations into
explicit, reusable insights and derives a target-specific reasoning trace;
optionally, the reasoning is self-refined for coherence and correctness (I2S+).
Extensive experiments on diverse benchmarks show that I2S and I2S+ consistently
outperform both direct answering and test-time scaling baselines across open-
and closed-source models. Even for GPT models, our method helps: on AIME'25,
GPT-4.1 rises by +14.0%, and o1-mini improves by +2.7% on AIME and +1.7% on
GPQA, indicating that in-context demonstrations can be harnessed effectively
via insight-refine-solve framework.
68db481cd2bf1f4b15ec73ec
Haonan Wang
68db481cd2bf1f4b15ec73ed
Weida Liang
68db481cd2bf1f4b15ec73ee
Zihang Fu
68db481cd2bf1f4b15ec73ef
Nie Zheng
68db481cd2bf1f4b15ec73f0
Yifan Zhang
68db481cd2bf1f4b15ec73f1
Yao Tong
68db481cd2bf1f4b15ec73f2
Tongyao Zhu
claimed_verified
2025-09-30T06:29:58.090Z
647db42711084fb58318b053
/avatars/901a4db8e99f6e0c7e7e596c251b315c.svg
Tongyao
user
tyzhu
68db481cd2bf1f4b15ec73f3
Hao Jiang
68db481cd2bf1f4b15ec73f4
Chuang Li
68db481cd2bf1f4b15ec73f5
Jiaying Wu
68db481cd2bf1f4b15ec73f6
Kenji Kawaguchi
From Harm to Help: Turning Reasoning In-Context Demos into Assets for
  Reasoning LMs
Recent reasoning LLMs (RLMs), especially those trained with verifier-based
reinforcement learning, often perform worse with few-shot CoT than with direct
answering. We revisit this paradox using high-quality reasoning traces from
DeepSeek-R1 as demonstrations and find that adding more exemplars consistently
degrades accuracy, even when demonstrations are optimal. A detailed analysis
reveals two mechanisms behind this decline: (i) semantic misguidance, where
high textual similarity leads the model to treat the target as the same as the
exemplar and to copy intermediate steps verbatim; and (ii) strategy transfer
failure, where the model struggles to extract useful reasoning strategies and
apply them to target questions. Guided by these, we introduce Insight-to-Solve
(I2S), a sequential test-time procedure that turns demonstrations into
explicit, reusable insights and derives a target-specific reasoning trace;
optionally, the reasoning is self-refined for coherence and correctness (I2S+).
Extensive experiments on diverse benchmarks show that I2S and I2S+ consistently
outperform both direct answering and test-time scaling baselines across open-
and closed-source models. Even for GPT models, our method helps: on AIME'25,
GPT-4.1 rises by +14.0%, and o1-mini improves by +2.7% on AIME and +1.7% on
GPQA, indicating that in-context demonstrations can be harnessed effectively
via insight-refine-solve framework.
Insight-to-Solve (I2S) and its refined version (I2S+) improve few-shot chain-of-thought performance by converting demonstrations into reusable insights, outperforming direct answering and scaling methods across various models.
reasoning LLMs verifier-based reinforcement learning few-shot CoT DeepSeek-R1 semantic misguidance strategy transfer failure Insight-to-Solve I2S I2S+ AIME'25 GPT-4.1 o1-mini GPQA