Reward-Guided Speculative Decoding for Efficient LLM Reasoning
We introduce Reward-Guided Speculative Decoding (RSD), a novel framework
aimed at improving the efficiency of inference in large language models (LLMs).
RSD synergistically combines a lightweight draft model with a more powerful
target model, incorporating a controlled bias to prioritize high-reward
outputs, in contrast to existing speculative decoding methods that enforce
strict unbiasedness. RSD employs a process reward model to evaluate
intermediate decoding steps and dynamically decide whether to invoke the target
model, optimizing the trade-off between computational cost and output quality.
We theoretically demonstrate that a threshold-based mixture strategy achieves
an optimal balance between resource utilization and performance. Extensive
evaluations on challenging reasoning benchmarks, including Olympiad-level
tasks, show that RSD delivers significant efficiency gains against decoding
with the target model only (up to 4.4x fewer FLOPs), while achieving
significant better accuracy than parallel decoding method on average (up to
+3.5). These results highlight RSD as a robust and cost-effective approach for
deploying LLMs in resource-intensive scenarios.
67a04151dd7b3a4aba880589
Baohao Liao
admin_assigned
2025-02-03T11:10:48.379Z
62c414354ce7250560a1f67f
/avatars/28fd73973d1703c84f4f59644fef8a80.svg
Baohao Liao
user
baohao
67a04151dd7b3a4aba88058a
Yuhui Xu
claimed_verified
2025-02-03T11:07:52.570Z
6602869253a0518b2a98cafd
/avatars/c14b5953a716f42c83ad28147f8308ae.svg
Yuhui Xu
user
yuhuixu
67a04151dd7b3a4aba88058b
Hanze Dong
admin_assigned
2025-02-03T11:10:54.880Z
63a3ff69f91ad3ea5703841d
/avatars/69227c4bce01d33747c1377b6f9672db.svg
Hanze Dong
user
hendrydong
67a04151dd7b3a4aba88058c
Junnan Li
admin_assigned
2025-02-03T11:11:08.170Z
61f9d3b54ac99e8a1bae85f4
/avatars/ac47d13204dd22452e4bc46e280842d5.svg
JunnanLi
user
JunnanLi
67a04151dd7b3a4aba88058d
Christof Monz
67a04151dd7b3a4aba88058e
Silvio Savarese
67a04151dd7b3a4aba88058f
Doyen Sahoo
admin_assigned
2025-02-03T11:11:23.345Z
65f84fd980481173afd91233
/avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
Doyen
user
doyensahoo
67a04151dd7b3a4aba880590
Caiming Xiong
admin_assigned
2025-02-03T11:11:29.821Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
We introduce Reward-Guided Speculative Decoding (RSD), a novel framework
aimed at improving the efficiency of inference in large language models (LLMs).
RSD synergistically combines a lightweight draft model with a more powerful
target model, incorporating a controlled bias to prioritize high-reward
outputs, in contrast to existing speculative decoding methods that enforce
strict unbiasedness. RSD employs a process reward model to evaluate
intermediate decoding steps and dynamically decide whether to invoke the target
model, optimizing the trade-off between computational cost and output quality.
We theoretically demonstrate that a threshold-based mixture strategy achieves
an optimal balance between resource utilization and performance. Extensive
evaluations on challenging reasoning benchmarks, including Olympiad-level
tasks, show that RSD delivers significant efficiency gains against decoding
with the target model only (up to 4.4x fewer FLOPs), while achieving
significant better accuracy than parallel decoding method on average (up to
+3.5). These results highlight RSD as a robust and cost-effective approach for
deploying LLMs in resource-intensive scenarios.
Reward-Guided Speculative Decoding improves inference efficiency in large language models by combining a draft and target model with a reward-based strategy, achieving better performance and cost savings.
Reward-Guided Speculative Decoding speculative decoding draft model target model process reward model threshold-based mixture strategy computational cost output quality FLOPs parallel decoding