FR-Spec: Accelerating Large-Vocabulary Language Models via
  Frequency-Ranked Speculative Sampling
Speculative sampling has emerged as an important technique for accelerating
the auto-regressive generation process of large language models (LLMs) by
utilizing a draft-then-verify mechanism to produce multiple tokens per forward
pass. While state-of-the-art speculative sampling methods use only a single
layer and a language modeling (LM) head as the draft model to achieve
impressive layer compression, their efficiency gains are substantially reduced
for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens.
To address this, we present FR-Spec, a frequency-ranked speculative sampling
framework that optimizes draft candidate selection through vocabulary space
compression. By constraining the draft search to a frequency-prioritized token
subset, our method reduces LM Head computation overhead by 75% while ensuring
the equivalence of the final output distribution. Experiments across multiple
datasets demonstrate an average of 1.12times speedup over the
state-of-the-art speculative sampling method EAGLE-2.
67bee83509a4524abf899511
Weilin Zhao
claimed_verified
2025-03-05T10:12:38.111Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
67bee83509a4524abf899512
Tengyu Pan
admin_assigned
2025-03-05T11:01:09.673Z
6391a84891023bed86fc703e
/avatars/6e61d845aded1542e61f5220365e0d6f.svg
ThonyPan
user
ThonyPan
67bee83509a4524abf899513
Xu Han
67bee83509a4524abf899514
Yudi Zhang
67bee83509a4524abf899515
Ao Sun
67bee83509a4524abf899516
Yuxiang Huang
claimed_verified
2025-03-05T10:12:36.389Z
65394f942ddab2beff1cc967
/avatars/bdd5b47618758f6de080a51834a17655.svg
Shawn Huang
user
hyx21
67bee83509a4524abf899517
Kaihuo Zhang
67bee83509a4524abf899518
Weilun Zhao
admin_assigned
2025-03-05T11:05:23.448Z
6716194e503429f9f5fb5116
/avatars/4707e293df6bdc14dc97a6b6f3b8932f.svg
zhao weilun
user
zwl96
67bee83509a4524abf899519
Yuxuan Li
admin_assigned
2025-03-05T11:05:30.760Z
66d86ae34eb2eb8dc078f808
/avatars/4fb926729092ce69087343ce767387c8.svg
Yuxuan Li
user
yuxuanli
67bee83509a4524abf89951a
Jianyong Wang
67bee83509a4524abf89951b
Zhiyuan Liu
admin_assigned
2025-03-05T11:06:05.421Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
67bee83509a4524abf89951c
Maosong Sun
FR-Spec: Accelerating Large-Vocabulary Language Models via
  Frequency-Ranked Speculative Sampling
Speculative sampling has emerged as an important technique for accelerating
the auto-regressive generation process of large language models (LLMs) by
utilizing a draft-then-verify mechanism to produce multiple tokens per forward
pass. While state-of-the-art speculative sampling methods use only a single
layer and a language modeling (LM) head as the draft model to achieve
impressive layer compression, their efficiency gains are substantially reduced
for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens.
To address this, we present FR-Spec, a frequency-ranked speculative sampling
framework that optimizes draft candidate selection through vocabulary space
compression. By constraining the draft search to a frequency-prioritized token
subset, our method reduces LM Head computation overhead by 75% while ensuring
the equivalence of the final output distribution. Experiments across multiple
datasets demonstrate an average of 1.12times speedup over the
state-of-the-art speculative sampling method EAGLE-2.
FR-Spec is a speculative sampling framework that improves speed by compressing the vocabulary space, reducing LM head computation overhead while maintaining output equivalence.
speculative sampling auto-regressive generation large language models draft-then-verify vocabulary space compression LM head frequency-ranked speculative sampling EAGLE-2