OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device
  Speculative Decoding
Speculative decoding generally dictates having a small, efficient draft model
that is either pretrained or distilled offline to a particular target model
series, for instance, Llama or Qwen models. However, within online deployment
settings, there are two major challenges: 1) usage of a target model that is
incompatible with the draft model; 2) expectation of latency improvements over
usage and time. In this work, we propose OmniDraft, a unified framework that
enables a single draft model to operate with any target model and adapt
dynamically to user data. We introduce an online n-gram cache with hybrid
distillation fine-tuning to address the cross-vocabulary mismatch across draft
and target models; and further improve decoding speed by leveraging adaptive
drafting techniques. OmniDraft is particularly suitable for on-device LLM
applications where model cost, efficiency and user customization are the major
points of contention. This further highlights the need to tackle the above
challenges and motivates the ``one drafter for all'' paradigm. We
showcase the proficiency of the OmniDraft framework by performing online
learning on math reasoning, coding and text generation tasks. Notably,
OmniDraft enables a single Llama-68M model to pair with various target models
including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding;
and additionally provides up to 1.5-2x speedup.
686736ed9db35afc9c304cea
Ramchalam Kinattinkara Ramakrishnan
claimed_verified
2025-07-08T13:53:01.025Z
6667a3058e3ed591818538e4
/avatars/0506605163b141cebb39a4092b3b6d83.svg
Ramchalam K R
user
ramchalamkr
686736ed9db35afc9c304ceb
Zhaocong Yuan
admin_assigned
2025-07-07T15:46:47.568Z
65d989790733541e06823258
/avatars/cf658c5f586c02e9b1d610b5e49b2826.svg
Zhaocong Yuan
user
justinyyy
686736ed9db35afc9c304cec
Shaojie Zhuo
claimed_verified
2025-07-28T08:46:40.602Z
639745275aba868df22b7e45
/avatars/0630fe335a15683a2674ac82113b7629.svg
Jay Zhuo
user
jayzhuo
686736ed9db35afc9c304ced
Chen Feng
claimed_verified
2025-07-09T08:51:42.004Z
673f8a1fb86fd98d9509e59a
/avatars/63873c99e05f18ecfc85a9ad0c4549d1.svg
Feng
user
Cindy427
686736ed9db35afc9c304cee
Yicheng Lin
686736ed9db35afc9c304cef
Chenzheng Su
claimed_verified
2025-07-08T14:06:04.617Z
686d24c697e15692bc44fa08
/avatars/3b4cc3dd9168525cc70b1599cc46219d.svg
Chenzheng Su
user
jokaysu
686736ed9db35afc9c304cf0
Xiaopeng Zhang
OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device
  Speculative Decoding
Speculative decoding generally dictates having a small, efficient draft model
that is either pretrained or distilled offline to a particular target model
series, for instance, Llama or Qwen models. However, within online deployment
settings, there are two major challenges: 1) usage of a target model that is
incompatible with the draft model; 2) expectation of latency improvements over
usage and time. In this work, we propose OmniDraft, a unified framework that
enables a single draft model to operate with any target model and adapt
dynamically to user data. We introduce an online n-gram cache with hybrid
distillation fine-tuning to address the cross-vocabulary mismatch across draft
and target models; and further improve decoding speed by leveraging adaptive
drafting techniques. OmniDraft is particularly suitable for on-device LLM
applications where model cost, efficiency and user customization are the major
points of contention. This further highlights the need to tackle the above
challenges and motivates the ``one drafter for all'' paradigm. We
showcase the proficiency of the OmniDraft framework by performing online
learning on math reasoning, coding and text generation tasks. Notably,
OmniDraft enables a single Llama-68M model to pair with various target models
including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding;
and additionally provides up to 1.5-2x speedup.
OmniDraft is a unified framework enabling a single draft model to dynamically adapt to various target models, addressing cross-vocabulary mismatches and improving decoding speed for on-device LLM applications.
speculative decoding draft model pretrained distilled target model OmniDraft online n-gram cache hybrid distillation fine-tuning adaptive drafting LLM Vicuna-7B Qwen2-7B Llama3-8B math reasoning coding text generation