Accelerating LLM Inference with Staged Speculative Decoding
Recent advances with large language models (LLM) illustrate their diverse
capabilities. We propose a novel algorithm, staged speculative decoding, to
accelerate LLM inference in small-batch, on-device scenarios. We address the
low arithmetic intensity of small-batch inference by improving upon previous
work in speculative decoding. First, we restructure the speculative batch as a
tree, which reduces generation costs and increases the expected tokens per
batch. Second, we add a second stage of speculative decoding. Taken together,
we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L
model while perfectly preserving output quality.
64d4498e0f3095b62d3b6440
Benjamin Spector
claimed_verified
2023-08-11T13:30:31.224Z
63d074db623a3d1d117372ad
/avatars/dbccd7cf1b2592bbc76dabcacf0a1e2f.svg
Benjamin Spector
user
benjaminfspector
64d4498e0f3095b62d3b6441
Chris Re
Accelerating LLM Inference with Staged Speculative Decoding
Recent advances with large language models (LLM) illustrate their diverse
capabilities. We propose a novel algorithm, staged speculative decoding, to
accelerate LLM inference in small-batch, on-device scenarios. We address the
low arithmetic intensity of small-batch inference by improving upon previous
work in speculative decoding. First, we restructure the speculative batch as a
tree, which reduces generation costs and increases the expected tokens per
batch. Second, we add a second stage of speculative decoding. Taken together,
we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L
model while perfectly preserving output quality.
A staged speculative decoding algorithm enhances LLM inference efficiency in small-batch, on-device settings by restructuring speculative batches and adding a second decoding stage.
large language models staged speculative decoding speculative decoding tree structure GPT-2-L parameter-efficient fine-tuning