Toward Evaluative Thinking: Meta Policy Optimization with Evolving
  Reward Models
Reward-based alignment methods for large language models (LLMs) face two key
limitations: vulnerability to reward hacking, where models exploit flaws in the
reward signal; and reliance on brittle, labor-intensive prompt engineering when
LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a
framework that addresses these challenges by integrating a meta-reward model
that dynamically refines the reward model's prompt throughout training. In MPO,
the meta-reward model monitors the evolving training context and continuously
adjusts the reward model's prompt to maintain high alignment, providing an
adaptive reward signal that resists exploitation by the policy. This
meta-learning approach promotes a more stable policy optimization, and greatly
reduces the need for manual reward prompt design. It yields performance on par
with or better than models guided by extensively hand-crafted reward prompts.
Furthermore, we show that MPO maintains its effectiveness across diverse tasks,
such as question answering and mathematical reasoning, without requiring
specialized reward designs. Beyond standard RLAIF, MPO's meta-learning
formulation is readily extensible to higher-level alignment frameworks.
Overall, this method addresses theoretical and practical challenges in
reward-based RL alignment for LLMs, paving the way for more robust and
adaptable alignment strategies. The code and models will be publicly shared.
68119750ff0764f3840a7f93
Zae Myung Kim
claimed_verified
2025-04-30T07:55:58.262Z
61e0c5053a1781f66b4e9aed
https://cdn-avatars.huggingface.co/v1/production/uploads/1642120523097-61e0c5053a1781f66b4e9aed.jpeg
Zae Myung Kim
user
zaemyung
68119750ff0764f3840a7f94
Chanwoo Park
claimed_verified
2025-05-26T14:41:49.528Z
63f4e18407640ba6fcf92870
/avatars/9392cc42754ba06b32297f6678f58216.svg
Chanwoo Park
user
Chanwoo97
68119750ff0764f3840a7f95
Vipul Raheja
admin_assigned
2025-04-30T10:03:14.631Z
60985a0547dc3dbf8a976607
/avatars/3c37bf4b7c9db83a46af7c473ee4eb86.svg
Vipul Raheja
user
machineteacher
68119750ff0764f3840a7f96
Dongyeop Kang
admin_assigned
2025-04-30T10:03:26.612Z
64356b40a4bd75c62cbc5926
/avatars/5f4c603464e9c8ad613a3a25fa4cacbf.svg
Dongyeop Kang
user
dykang
Toward Evaluative Thinking: Meta Policy Optimization with Evolving
  Reward Models
Reward-based alignment methods for large language models (LLMs) face two key
limitations: vulnerability to reward hacking, where models exploit flaws in the
reward signal; and reliance on brittle, labor-intensive prompt engineering when
LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a
framework that addresses these challenges by integrating a meta-reward model
that dynamically refines the reward model's prompt throughout training. In MPO,
the meta-reward model monitors the evolving training context and continuously
adjusts the reward model's prompt to maintain high alignment, providing an
adaptive reward signal that resists exploitation by the policy. This
meta-learning approach promotes a more stable policy optimization, and greatly
reduces the need for manual reward prompt design. It yields performance on par
with or better than models guided by extensively hand-crafted reward prompts.
Furthermore, we show that MPO maintains its effectiveness across diverse tasks,
such as question answering and mathematical reasoning, without requiring
specialized reward designs. Beyond standard RLAIF, MPO's meta-learning
formulation is readily extensible to higher-level alignment frameworks.
Overall, this method addresses theoretical and practical challenges in
reward-based RL alignment for LLMs, paving the way for more robust and
adaptable alignment strategies. The code and models will be publicly shared.
MPO is a framework that dynamically refines reward signals in LLMs through meta-reward modeling, enhancing alignment and reducing reward hacking and prompt engineering needs.
reward-based alignment large language models reward hacking prompt engineering Meta Policy Optimization meta-reward model policy optimization reward-based RL alignment