POSS: Position Specialist Generates Better Draft for Speculative
  Decoding
Speculative decoding accelerates Large Language Model (LLM) inference by
using a small draft model to predict multiple tokens, and a large target model
to verify these tokens in parallel. Recent studies leverage the hidden state of
the target model to enhance draft model prediction accuracy. However, existing
methods suffer from the degrading quality of draft token predictions at later
positions, due to error accumulation in draft model generated features. In this
paper, we propose Position Specialists (PosS), which consist of multiple
position-specialized draft layers to generate tokens at assigned position(s).
Position specialists greatly improve token acceptance rate at later positions
per drafting round, as each specialist only needs to focus on handling a
certain level of draft model feature deviation. Experiment results on
Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that
PosS effectively improves over baselines on average acceptance length and
speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.
6841db4b1a53c86e5eceae97
Langlin Huang
claimed_verified
2025-06-06T07:41:48.349Z
65e02d89574e5aa0e9ce3efa
/avatars/2ab152a10b21d81fb1defc726b8e951a.svg
Langlin Huang
user
shrango
6841db4b1a53c86e5eceae98
Chengsong Huang
claimed_verified
2025-06-09T10:12:36.757Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
user
ChengsongHuang
6841db4b1a53c86e5eceae99
Jixuan Leng
6841db4b1a53c86e5eceae9a
Di Huang
6841db4b1a53c86e5eceae9b
Jiaxin Huang
POSS: Position Specialist Generates Better Draft for Speculative
  Decoding
Speculative decoding accelerates Large Language Model (LLM) inference by
using a small draft model to predict multiple tokens, and a large target model
to verify these tokens in parallel. Recent studies leverage the hidden state of
the target model to enhance draft model prediction accuracy. However, existing
methods suffer from the degrading quality of draft token predictions at later
positions, due to error accumulation in draft model generated features. In this
paper, we propose Position Specialists (PosS), which consist of multiple
position-specialized draft layers to generate tokens at assigned position(s).
Position specialists greatly improve token acceptance rate at later positions
per drafting round, as each specialist only needs to focus on handling a
certain level of draft model feature deviation. Experiment results on
Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that
PosS effectively improves over baselines on average acceptance length and
speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.
Position Specialists (PosS) enhance Large Language Model (LLM) inference by using position-specialized draft layers to improve token prediction accuracy and acceptance rates.
Speculative decoding LLM Large Language Model draft model target model token prediction Position Specialists PosS token acceptance rate feature deviation Llama-3-8B-Instruct Llama-2-13B-chat acceptance length speed-up ratio