Ouroboros: Speculative Decoding with Large Model Enhanced Drafting
Drafting-then-verifying decoding methods such as speculative decoding are
widely adopted training-free methods to accelerate the inference of large
language models (LLMs). Instead of employing an autoregressive process to
decode tokens sequentially, speculative decoding initially creates drafts with
an efficient small model. Then LLMs are required to conduct verification and
correction in a non-autoregressive fashion to minimize time overhead.
Generating longer drafts can lead to even more significant speedups once
verified, but also incurs substantial trial and error costs if it fails.
Suffering from the high verification failure probability, existing decoding
methods cannot draft too much content for verification at one time, achieving
sub-optimal inference acceleration. In this paper, we introduce Ouroboros,
which constructs a phrase candidate pool from the verification process of LLMs
to provide candidates for draft generation of the small model. Thereby,
Ouroboros can further improve the efficiency and effectiveness of the initial
drafts. The experimental results on typical text generation tasks show that
Ouroboros achieves speedups of up to 1.9x and 2.8x compared to lookahead
decoding and speculative decoding, respectively. The source code of Ouroboros
is available at https://github.com/thunlp/Ouroboros.
65d6cb3258e71c5328879180
Weilin Zhao
claimed_verified
2025-03-05T10:13:16.980Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
65d6cb3258e71c5328879181
Yuxiang Huang
claimed_verified
2025-03-05T10:13:15.173Z
65394f942ddab2beff1cc967
/avatars/bdd5b47618758f6de080a51834a17655.svg
Shawn Huang
user
hyx21
65d6cb3258e71c5328879182
Xu Han
65d6cb3258e71c5328879183
Chaojun Xiao
claimed_verified
2024-09-09T06:57:49.844Z
608f6d72283d0a8d7be9d1f9
/avatars/7f499a37019359a3c488ba6cc11751fc.svg
Chaojun XIAO
user
xcjthu
65d6cb3258e71c5328879184
Zhiyuan Liu
claimed_verified
2025-06-02T08:00:26.231Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
65d6cb3258e71c5328879185
Maosong Sun
Ouroboros: Speculative Decoding with Large Model Enhanced Drafting
Drafting-then-verifying decoding methods such as speculative decoding are
widely adopted training-free methods to accelerate the inference of large
language models (LLMs). Instead of employing an autoregressive process to
decode tokens sequentially, speculative decoding initially creates drafts with
an efficient small model. Then LLMs are required to conduct verification and
correction in a non-autoregressive fashion to minimize time overhead.
Generating longer drafts can lead to even more significant speedups once
verified, but also incurs substantial trial and error costs if it fails.
Suffering from the high verification failure probability, existing decoding
methods cannot draft too much content for verification at one time, achieving
sub-optimal inference acceleration. In this paper, we introduce Ouroboros,
which constructs a phrase candidate pool from the verification process of LLMs
to provide candidates for draft generation of the small model. Thereby,
Ouroboros can further improve the efficiency and effectiveness of the initial
drafts. The experimental results on typical text generation tasks show that
Ouroboros achieves speedups of up to 1.9x and 2.8x compared to lookahead
decoding and speculative decoding, respectively. The source code of Ouroboros
is available at https://github.com/thunlp/Ouroboros.
Ouroboros, a drafting-then-verifying decoding method, enhances efficiency by using a phrase candidate pool for draft generation, achieving significant speedups over lookahead and speculative decoding.
speculative decoding large language models autoregressive process non-autoregressive lookahead decoding phrase candidate pool