When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs
Recent Long-Context Language Models (LCLMs) can process hundreds of thousands
of tokens in a single prompt, enabling new opportunities for
knowledge-intensive multi-hop reasoning by integrating large sets of retrieved
documents or, in some cases, directly all necessary information. However,
simply feeding more documents into the context window fails to capture how
evidence should be connected. We address this gap with thought templates, which
recast reasoning as reusable thought caches, derived from prior problem solving
traces, structuring how evidence is combined and guiding multi-hop inference
with factual documents. To keep these templates effective, we propose an update
strategy that iteratively refines templates derived from training data through
natural-language feedback. Across diverse benchmarks and LCLM families, our
approach delivers consistent gains over strong baselines in both
retrieval-based and retrieval-free settings. Furthermore, we show that
optimized templates can be distilled into smaller open-source models,
demonstrating its broad applicability and transparent reasoning reuse. We refer
to our framework as Thought Template Augmented LCLMs (ToTAL).
68e881c695e8e6771df38a69
Soyeong Jeong
claimed_verified
2025-10-10T08:55:34.734Z
64e5a1cd4c20016ec9020ec8
/avatars/d7ffe7fbbe39c0a013375357457c57b3.svg
Soyeong
user
starsuzi
68e881c695e8e6771df38a6a
Taehee Jung
claimed_verified
2025-10-11T13:46:07.872Z
6626f092b4d4fbf85f421a81
/avatars/488a06fef21dc188e9d9240f3724501b.svg
Taehee Jung
user
theejung
68e881c695e8e6771df38a6b
Sung Ju Hwang
68e881c695e8e6771df38a6c
Joo-Kyung Kim
68e881c695e8e6771df38a6d
Dongyeop Kang
When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs
Recent Long-Context Language Models (LCLMs) can process hundreds of thousands
of tokens in a single prompt, enabling new opportunities for
knowledge-intensive multi-hop reasoning by integrating large sets of retrieved
documents or, in some cases, directly all necessary information. However,
simply feeding more documents into the context window fails to capture how
evidence should be connected. We address this gap with thought templates, which
recast reasoning as reusable thought caches, derived from prior problem solving
traces, structuring how evidence is combined and guiding multi-hop inference
with factual documents. To keep these templates effective, we propose an update
strategy that iteratively refines templates derived from training data through
natural-language feedback. Across diverse benchmarks and LCLM families, our
approach delivers consistent gains over strong baselines in both
retrieval-based and retrieval-free settings. Furthermore, we show that
optimized templates can be distilled into smaller open-source models,
demonstrating its broad applicability and transparent reasoning reuse. We refer
to our framework as Thought Template Augmented LCLMs (ToTAL).
Thought templates enhance long-context language models by structuring evidence combination and guiding multi-hop inference, leading to consistent performance improvements across various benchmarks.
Long-Context Language Models thought templates reusable thought caches multi-hop reasoning evidence combination natural-language feedback retrieval-based retrieval-free ToTAL