Speculative Streaming: Fast LLM Inference without Auxiliary Models
Speculative decoding is a prominent technique to speed up the inference of a
large target language model based on predictions of an auxiliary draft model.
While effective, in application-specific settings, it often involves
fine-tuning both draft and target models to achieve high acceptance rates. As
the number of downstream tasks grows, these draft models add significant
complexity to inference systems. We propose Speculative Streaming, a
single-model speculative decoding method that fuses drafting into the target
model by changing the fine-tuning objective from next token prediction to
future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 -
3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and
Meaning Representation, without sacrificing generation quality. Additionally,
Speculative Streaming is parameter-efficient. It achieves on-par/higher
speed-ups than Medusa-style architectures while using ~10000X fewer extra
parameters, making it well-suited for resource-constrained devices.
65d438710cf70a406c627722
Nikhil Bhendawade
claimed_verified
2024-02-21T08:30:34.595Z
64d5061ea146b1c0a63f91c6
/avatars/41f322d00c859cd345b853b55a3ad59f.svg
Nikhil
user
NickNickGo
65d438710cf70a406c627723
Irina Belousova
claimed_verified
2024-02-20T18:24:19.029Z
65d4e7cbc81e8c3cffafa8af
/avatars/aa490b7e103320c8951b1c3beb242e1d.svg
Irina Belousova
user
ibelousova
65d438710cf70a406c627724
Qichen Fu
admin_assigned
2024-02-20T09:45:37.162Z
6496114d33611fb433f16970
/avatars/ac722222c39e87c772705a3bf52118ad.svg
Qichen Fu
user
janemiminini
65d438710cf70a406c627725
Henry Mason
claimed_verified
2024-02-20T18:24:21.752Z
630d189254c3dbd48056bf63
/avatars/6b7a3dd235be1ba5679775ca53c4116d.svg
Henry Mason
user
docgok
65d438710cf70a406c627726
Mohammad Rastegari
65d438710cf70a406c627727
Mahyar Najibi
extracted_confirmed
2024-04-15T21:44:36.448Z
661d992d69a65c0fb0717013
/avatars/9d1f20baff742de90a50232db54d2407.svg
Mahyar Najibi
user
mahyar-najibi
Speculative Streaming: Fast LLM Inference without Auxiliary Models
Speculative decoding is a prominent technique to speed up the inference of a
large target language model based on predictions of an auxiliary draft model.
While effective, in application-specific settings, it often involves
fine-tuning both draft and target models to achieve high acceptance rates. As
the number of downstream tasks grows, these draft models add significant
complexity to inference systems. We propose Speculative Streaming, a
single-model speculative decoding method that fuses drafting into the target
model by changing the fine-tuning objective from next token prediction to
future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 -
3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and
Meaning Representation, without sacrificing generation quality. Additionally,
Speculative Streaming is parameter-efficient. It achieves on-par/higher
speed-ups than Medusa-style architectures while using ~10000X fewer extra
parameters, making it well-suited for resource-constrained devices.
Speculative Streaming is a method that incorporates drafting into a target language model, enhancing decoding speed without quality loss while using fewer parameters.
speculative decoding large target language model auxiliary draft model fine-tuning next token prediction future n-gram prediction speculative streaming decoding speed Summarization Structured Queries Meaning Representation parameter-efficient Medusa-style architectures