Lean and Mean: Decoupled Value Policy Optimization with Global Value
  Guidance
Proximal Policy Optimization (PPO)-based Reinforcement Learning from Human
Feedback (RLHF) is essential for aligning large language models (LLMs) with
human preferences. It requires joint training of an actor and critic with a
pretrained, fixed reward model for guidance. This approach increases
computational complexity and instability due to actor-critic interdependence.
Additionally, PPO lacks access to true environment rewards in LLM tasks,
limiting its adaptability. Under such conditions, pretraining a value model or
a reward model becomes equivalent, as both provide fixed supervisory signals
without new ground-truth feedback. To address these issues, we propose
Decoupled Value Policy Optimization (DVPO), a lean framework that
replaces traditional reward modeling with a pretrained global value model
(GVM). The GVM is conditioned on policy trajectories and predicts token-level
return-to-go estimates. By decoupling value model from policy training (via
frozen GVM-driven RL objectives), DVPO eliminates actor-critic interdependence,
reducing GPU memory usage by 40\% and training time by 35\% compared to
conventional RLHF. Experiments across benchmarks show DVPO outperforms
efficient RLHF methods (e.g., DPO) while matching state-of-the-art PPO in
performance.
67be807e8a5a805423137ca2
Chenghua Huang
admin_assigned
2025-02-28T12:30:31.972Z
664af07a691370727c281031
/avatars/e5ed17342e0ea953bacc7d57e9f3b686.svg
Cheng Hua Huang
user
LanceZomax
67be807e8a5a805423137ca3
Lu Wang
admin_assigned
2025-02-28T12:30:39.129Z
6406afa8a577649430c64363
/avatars/9bd1768c91d509c8c49970e9fd7775a5.svg
LuWang
user
LuWang
67be807e8a5a805423137ca4
Fangkai Yang
claimed_verified
2025-02-27T09:17:46.382Z
669dcf6200970c3b27aafa5d
/avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
kaikai yang
user
keanudicap
67be807e8a5a805423137ca5
Pu Zhao
67be807e8a5a805423137ca6
Zhixu Li
admin_assigned
2025-02-28T12:30:46.084Z
661dd71d2ae9013218415e6f
/avatars/389883e5886628c07cb0b08fc8c93c3b.svg
Zhixu Li
user
ZhixuLi
67be807e8a5a805423137ca7
Qingwei Lin
admin_assigned
2025-02-28T12:31:00.637Z
652fc9f39bc50a6c0e435224
https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
Lin Qingwei
user
Eliblo1969
67be807e8a5a805423137ca8
Dongmei Zhang
admin_assigned
2025-02-28T12:31:14.045Z
66473d2c7abe6ad66e81a3dd
/avatars/82f40244806c06ffeaa1c4265e9725ea.svg
ZHANGDONGMEI
user
ZDM6426
67be807e8a5a805423137ca9
Saravan Rajmohan
67be807e8a5a805423137caa
Qi Zhang
Lean and Mean: Decoupled Value Policy Optimization with Global Value
  Guidance
Proximal Policy Optimization (PPO)-based Reinforcement Learning from Human
Feedback (RLHF) is essential for aligning large language models (LLMs) with
human preferences. It requires joint training of an actor and critic with a
pretrained, fixed reward model for guidance. This approach increases
computational complexity and instability due to actor-critic interdependence.
Additionally, PPO lacks access to true environment rewards in LLM tasks,
limiting its adaptability. Under such conditions, pretraining a value model or
a reward model becomes equivalent, as both provide fixed supervisory signals
without new ground-truth feedback. To address these issues, we propose
Decoupled Value Policy Optimization (DVPO), a lean framework that
replaces traditional reward modeling with a pretrained global value model
(GVM). The GVM is conditioned on policy trajectories and predicts token-level
return-to-go estimates. By decoupling value model from policy training (via
frozen GVM-driven RL objectives), DVPO eliminates actor-critic interdependence,
reducing GPU memory usage by 40\% and training time by 35\% compared to
conventional RLHF. Experiments across benchmarks show DVPO outperforms
efficient RLHF methods (e.g., DPO) while matching state-of-the-art PPO in
performance.
Decoupled Value Policy Optimization (DVPO) enhances Reinforcement Learning from Human Feedback by using a pretrained global value model, improving performance and reducing computational requirements compared to existing methods.
Proximal Policy Optimization Reinforcement Learning from Human Feedback large language models actor-critic pretraining global value model token-level return-to-go DVPO decoupled value model RLHF