Iterative Value Function Optimization for Guided Decoding
While Reinforcement Learning from Human Feedback (RLHF) has become the
predominant method for controlling language model outputs, it suffers from high
computational costs and training instability. Guided decoding, especially
value-guided methods, offers a cost-effective alternative by controlling
outputs without re-training models. However, the accuracy of the value function
is crucial for value-guided decoding, as inaccuracies can lead to suboptimal
decision-making and degraded performance. Existing methods struggle with
accurately estimating the optimal value function, leading to less effective
control. We propose Iterative Value Function Optimization, a novel framework
that addresses these limitations through two key components: Monte Carlo Value
Estimation, which reduces estimation variance by exploring diverse
trajectories, and Iterative On-Policy Optimization, which progressively
improves value estimation through collecting trajectories from value-guided
policies. Extensive experiments on text summarization, multi-turn dialogue, and
instruction following demonstrate the effectiveness of value-guided decoding
approaches in aligning language models. These approaches not only achieve
alignment but also significantly reduce computational costs by leveraging
principled value function optimization for efficient and effective control.
67c80f94ccc2e04adfa67079
Zhenhua Liu
admin_assigned
2025-03-05T10:42:35.402Z
619b00dd4b0db5ca9d3ea35f
/avatars/fce5ac388b7f10cbbc63e9992a5a799f.svg
Zhenhua Liu
user
zhliu
67c80f94ccc2e04adfa6707a
Lijun Li
admin_assigned
2025-03-05T10:41:31.679Z
65ca1fbbecfd8608f5b89666
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/YM6yIAjVQCRHjk6uJd1jr.jpeg
lilijun
user
lljhbxt
67c80f94ccc2e04adfa6707b
Ruizhe Chen
67c80f94ccc2e04adfa6707c
Yuxian Jiang
admin_assigned
2025-03-05T10:40:51.943Z
64a67bff9f3b568c203cb533
/avatars/630d117a6e4392b78b35d6b95066b52a.svg
YU-XIAN JIANG
user
ab3223323
67c80f94ccc2e04adfa6707d
Tong Zhu
67c80f94ccc2e04adfa6707e
Wenliang Chen
admin_assigned
2025-03-05T10:40:45.604Z
63477237511cd17d2c7a1be9
/avatars/97cb3f640ded58344bca0ab279473134.svg
Wenliang Chen
user
jokephp
67c80f94ccc2e04adfa6707f
Jing Shao
Iterative Value Function Optimization for Guided Decoding
While Reinforcement Learning from Human Feedback (RLHF) has become the
predominant method for controlling language model outputs, it suffers from high
computational costs and training instability. Guided decoding, especially
value-guided methods, offers a cost-effective alternative by controlling
outputs without re-training models. However, the accuracy of the value function
is crucial for value-guided decoding, as inaccuracies can lead to suboptimal
decision-making and degraded performance. Existing methods struggle with
accurately estimating the optimal value function, leading to less effective
control. We propose Iterative Value Function Optimization, a novel framework
that addresses these limitations through two key components: Monte Carlo Value
Estimation, which reduces estimation variance by exploring diverse
trajectories, and Iterative On-Policy Optimization, which progressively
improves value estimation through collecting trajectories from value-guided
policies. Extensive experiments on text summarization, multi-turn dialogue, and
instruction following demonstrate the effectiveness of value-guided decoding
approaches in aligning language models. These approaches not only achieve
alignment but also significantly reduce computational costs by leveraging
principled value function optimization for efficient and effective control.
A novel framework, Iterative Value Function Optimization, improves value-guided decoding in reinforcement learning by enhancing value function estimation accuracy, aligning language models and reducing computational costs.
Reinforcement Learning from Human Feedback (RLHF) value-guided methods value function Monte Carlo Value Estimation Iterative On-Policy Optimization text summarization multi-turn dialogue instruction following