AdaSPEC: Selective Knowledge Distillation for Efficient Speculative
  Decoders
Speculative Decoding (SD) accelerates large language model inference by
employing a small draft model to generate predictions, which are then verified
by a larger target model. The effectiveness of SD hinges on the alignment
between these models, which is typically enhanced by Knowledge Distillation
(KD). However, conventional KD methods aim to minimize the KL divergence
between the draft and target models across all tokens, a goal that is
misaligned with the true objective of SD, which is to maximize token acceptance
rate. Therefore, draft models often struggle to fully assimilate the target
model's knowledge due to capacity constraints, leading to suboptimal
performance. To address this challenge, we propose AdaSPEC, a novel method that
incorporates selective token filtering into the KD process. AdaSPEC utilizes a
reference model to identify and filter out difficult-to-fit tokens, enabling
the distillation of a draft model that better aligns with the target model on
simpler tokens. This approach improves the overall token acceptance rate
without compromising generation quality. We evaluate AdaSPEC across diverse
tasks, including arithmetic reasoning, instruction-following, coding, and
summarization, using model configurations of 31M/1.4B and 350M/2.7B parameters.
Our results demonstrate that AdaSPEC consistently outperforms the
state-of-the-art DistillSpec method, achieving higher acceptance rates across
all tasks (up to 15\%). The code is publicly available at
https://github.com/yuezhouhu/adaspec.
68f983f6b9b2e4ae04673741
Yuezhou Hu
claimed_verified
2025-10-23T14:51:20.772Z
67dc66fe55c24fc4f981a4ab
/avatars/7bd900ade802d99db7c562ad6c2f6661.svg
Yuezhou Hu
user
yuezhouhu
68f983f6b9b2e4ae04673742
Jiaxin Guo
claimed_verified
2025-10-24T16:13:13.700Z
668ff1aed7741cade00afd40
https://cdn-avatars.huggingface.co/v1/production/uploads/668ff1aed7741cade00afd40/Pt8vYTL6KtcY0f3KJpymX.png
Jiaxin Guo
user
xinyuerufei
68f983f6b9b2e4ae04673743
Xinyu Feng
68f983f6b9b2e4ae04673744
Tuo Zhao
AdaSPEC: Selective Knowledge Distillation for Efficient Speculative
  Decoders
Speculative Decoding (SD) accelerates large language model inference by
employing a small draft model to generate predictions, which are then verified
by a larger target model. The effectiveness of SD hinges on the alignment
between these models, which is typically enhanced by Knowledge Distillation
(KD). However, conventional KD methods aim to minimize the KL divergence
between the draft and target models across all tokens, a goal that is
misaligned with the true objective of SD, which is to maximize token acceptance
rate. Therefore, draft models often struggle to fully assimilate the target
model's knowledge due to capacity constraints, leading to suboptimal
performance. To address this challenge, we propose AdaSPEC, a novel method that
incorporates selective token filtering into the KD process. AdaSPEC utilizes a
reference model to identify and filter out difficult-to-fit tokens, enabling
the distillation of a draft model that better aligns with the target model on
simpler tokens. This approach improves the overall token acceptance rate
without compromising generation quality. We evaluate AdaSPEC across diverse
tasks, including arithmetic reasoning, instruction-following, coding, and
summarization, using model configurations of 31M/1.4B and 350M/2.7B parameters.
Our results demonstrate that AdaSPEC consistently outperforms the
state-of-the-art DistillSpec method, achieving higher acceptance rates across
all tasks (up to 15\%). The code is publicly available at
https://github.com/yuezhouhu/adaspec.
AdaSPEC enhances speculative decoding by selectively filtering tokens during knowledge distillation, improving token acceptance rates without sacrificing generation quality.
Speculative Decoding Knowledge Distillation KL divergence token acceptance rate selective token filtering reference model arithmetic reasoning instruction-following coding summarization