Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads
  to Answers Faster
In this work, we propose FastCoT, a model-agnostic framework based on
parallel decoding without any further training of an auxiliary model or
modification to the LLM itself. FastCoT uses a size-varying context window
whose size changes with position to conduct parallel decoding and
auto-regressive decoding simultaneously, thus fully utilizing GPU computation
resources. In FastCoT, the parallel decoding part provides the LLM with a quick
glance of the future composed of approximate tokens, which could lead to faster
answers compared to regular autoregressive decoding used by causal
transformers. We also provide an implementation of parallel decoding within
LLM, which supports KV-cache generation and batch processing. Through extensive
experiments, we demonstrate that FastCoT saves inference time by nearly 20%
with only a negligible performance drop compared to the regular approach.
Additionally, we show that the context window size exhibits considerable
robustness for different tasks.
655446884045e53f450414d3
Hongxuan Zhang
655446884045e53f450414d4
Zhining Liu
admin_assigned
2023-11-15T10:02:37.504Z
61362d107c6d9b8e41469bda
/avatars/b0588156bb9da87f05ed89dc9cf0a378.svg
Zhining Liu
user
Zhin
655446884045e53f450414d5
Jiaqi Zheng
655446884045e53f450414d6
Chenyi Zhuang
claimed_verified
2025-08-11T06:46:59.066Z
64e847ab5ddcace745b8f5b1
/avatars/89525e66ff2900f86f66a11043a298f9.svg
chenyi zhuang
user
chengle
655446884045e53f450414d7
Jinjie Gu
655446884045e53f450414d8
Guihai Chen
Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads
  to Answers Faster
In this work, we propose FastCoT, a model-agnostic framework based on
parallel decoding without any further training of an auxiliary model or
modification to the LLM itself. FastCoT uses a size-varying context window
whose size changes with position to conduct parallel decoding and
auto-regressive decoding simultaneously, thus fully utilizing GPU computation
resources. In FastCoT, the parallel decoding part provides the LLM with a quick
glance of the future composed of approximate tokens, which could lead to faster
answers compared to regular autoregressive decoding used by causal
transformers. We also provide an implementation of parallel decoding within
LLM, which supports KV-cache generation and batch processing. Through extensive
experiments, we demonstrate that FastCoT saves inference time by nearly 20%
with only a negligible performance drop compared to the regular approach.
Additionally, we show that the context window size exhibits considerable
robustness for different tasks.
FastCoT, a model-agnostic framework, leverages parallel decoding with a dynamic context window to reduce inference time for large language models without significant performance degradation.
FastCoT parallel decoding context window LLM auto-regressive decoding causal transformers KV-cache batch processing