SpecReason: Fast and Accurate Inference-Time Compute via Speculative
  Reasoning
Recent advances in inference-time compute have significantly improved
performance on complex tasks by generating long chains of thought (CoTs) using
Large Reasoning Models (LRMs). However, this improved accuracy comes at the
cost of high inference latency due to the length of generated reasoning
sequences and the autoregressive nature of decoding. Our key insight in
tackling these overheads is that LRM inference, and the reasoning that it
embeds, is highly tolerant of approximations: complex tasks are typically
broken down into simpler steps, each of which brings utility based on the
semantic insight it provides for downstream steps rather than the exact tokens
it generates. Accordingly, we introduce SpecReason, a system that automatically
accelerates LRM inference by using a lightweight model to (speculatively) carry
out simpler intermediate reasoning steps and reserving the costly base model
only to assess (and potentially correct) the speculated outputs. Importantly,
SpecReason's focus on exploiting the semantic flexibility of thinking tokens in
preserving final-answer accuracy is complementary to prior speculation
techniques, most notably speculative decoding, which demands token-level
equivalence at each step. Across a variety of reasoning benchmarks, SpecReason
achieves 1.5-2.5times speedup over vanilla LRM inference while improving
accuracy by 1.0-9.9\%. Compared to speculative decoding without SpecReason,
their combination yields an additional 19.4-44.2\% latency reduction. We
open-source SpecReason at https://github.com/ruipeterpan/specreason.
67fdaecd1c5d730135243f5c
Rui Pan
claimed_verified
2025-04-15T07:55:03.136Z
62221f7af14f39d27896de15
/avatars/6318269df8c77e51f16445c29d65307c.svg
Rui Pan
user
ruipeterpan
67fdaecd1c5d730135243f5d
Yinwei Dai
67fdaecd1c5d730135243f5e
Zhihao Zhang
extracted_pending
2025-04-15T00:56:48.883Z
6480ec2d856901b0edbbd47a
/avatars/dd8c22589c46fc19d7bf4a1329692d91.svg
Zhihao Jia
user
zhihaojia
67fdaecd1c5d730135243f5f
Gabriele Oliaro
67fdaecd1c5d730135243f60
Zhihao Jia
67fdaecd1c5d730135243f61
Ravi Netravali
SpecReason: Fast and Accurate Inference-Time Compute via Speculative
  Reasoning
Recent advances in inference-time compute have significantly improved
performance on complex tasks by generating long chains of thought (CoTs) using
Large Reasoning Models (LRMs). However, this improved accuracy comes at the
cost of high inference latency due to the length of generated reasoning
sequences and the autoregressive nature of decoding. Our key insight in
tackling these overheads is that LRM inference, and the reasoning that it
embeds, is highly tolerant of approximations: complex tasks are typically
broken down into simpler steps, each of which brings utility based on the
semantic insight it provides for downstream steps rather than the exact tokens
it generates. Accordingly, we introduce SpecReason, a system that automatically
accelerates LRM inference by using a lightweight model to (speculatively) carry
out simpler intermediate reasoning steps and reserving the costly base model
only to assess (and potentially correct) the speculated outputs. Importantly,
SpecReason's focus on exploiting the semantic flexibility of thinking tokens in
preserving final-answer accuracy is complementary to prior speculation
techniques, most notably speculative decoding, which demands token-level
equivalence at each step. Across a variety of reasoning benchmarks, SpecReason
achieves 1.5-2.5times speedup over vanilla LRM inference while improving
accuracy by 1.0-9.9\%. Compared to speculative decoding without SpecReason,
their combination yields an additional 19.4-44.2\% latency reduction. We
open-source SpecReason at https://github.com/ruipeterpan/specreason.
SpecReason accelerates Large Reasoning Model inference by using a lightweight model for intermediate reasoning steps, improving speed without sacrificing accuracy.
SpecReason Large Reasoning Models inference latency reasoning sequences autoregressive decoding speculative decoding token-level equivalence