DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with
  Dynamic Multi-Sequence Drafting
Large language models (LLMs) exhibit exceptional performance across a wide
range of tasks; however, their token-by-token autoregressive generation process
significantly hinders inference speed. Speculative decoding presents a
promising draft-then-verify framework that reduces generation latency while
maintaining output distribution fidelity. Nevertheless, the draft model
introduces additional computational overhead, becoming a performance bottleneck
and increasing the time to first token (TTFT). Previous approaches to mitigate
draft model overhead have primarily relied on heuristics and generally failed
to match the quality of the draft language models. To address these challenges,
we propose DuoDecoding, a novel approach that strategically deploys the draft
and target models on the CPU and GPU respectively, enabling parallel decoding
while preserving draft quality. Our method incorporates a hardware-aware
optimal draft budget to minimize idle times and employs dynamic multi-sequence
drafting to enhance draft quality. Extensive experiments across seven tasks
show that DuoDecoding achieves up to 2.61x speedup in generation latency, while
reducing TTFT to 83% of that in conventional speculative decoding. The Code is
available at https://github.com/KaiLv69/DuoDecoding.
67c673bcf47209364f0cec96
Kai Lv
admin_assigned
2025-03-04T10:14:11.523Z
6485d5b300c9cfe5c2470c81
/avatars/c29aa81d2add795e8448b99274a04b83.svg
Kai
user
KaiLv
67c673bcf47209364f0cec97
Honglin Guo
admin_assigned
2025-03-04T10:14:04.672Z
638ef0b0c67af472d31674a6
/avatars/02df97d15a0f46b47f9162221733b121.svg
Honglin Guo
user
KYLN24
67c673bcf47209364f0cec98
Qipeng Guo
admin_assigned
2025-03-04T10:13:46.322Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
user
QipengGuo
67c673bcf47209364f0cec99
Xipeng Qiu
admin_assigned
2025-03-04T10:13:40.885Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with
  Dynamic Multi-Sequence Drafting
Large language models (LLMs) exhibit exceptional performance across a wide
range of tasks; however, their token-by-token autoregressive generation process
significantly hinders inference speed. Speculative decoding presents a
promising draft-then-verify framework that reduces generation latency while
maintaining output distribution fidelity. Nevertheless, the draft model
introduces additional computational overhead, becoming a performance bottleneck
and increasing the time to first token (TTFT). Previous approaches to mitigate
draft model overhead have primarily relied on heuristics and generally failed
to match the quality of the draft language models. To address these challenges,
we propose DuoDecoding, a novel approach that strategically deploys the draft
and target models on the CPU and GPU respectively, enabling parallel decoding
while preserving draft quality. Our method incorporates a hardware-aware
optimal draft budget to minimize idle times and employs dynamic multi-sequence
drafting to enhance draft quality. Extensive experiments across seven tasks
show that DuoDecoding achieves up to 2.61x speedup in generation latency, while
reducing TTFT to 83% of that in conventional speculative decoding. The Code is
available at https://github.com/KaiLv69/DuoDecoding.
DuoDecoding optimizes speculative decoding by strategically deploying draft and target models to parallelize processes, reducing generation latency and time to first token.
LLMs autoregressive generation speculative decoding draft model target model CPU GPU parallel decoding hardware-aware optimal draft budget dynamic multi-sequence drafting TTFT