Towards Faithful and Controllable Personalization via Critique-Post-Edit
  Reinforcement Learning
Faithfully personalizing large language models (LLMs) to align with
individual user preferences is a critical but challenging task. While
supervised fine-tuning (SFT) quickly reaches a performance plateau, standard
reinforcement learning from human feedback (RLHF) also struggles with the
nuances of personalization. Scalar-based reward models are prone to reward
hacking which leads to verbose and superficially personalized responses. To
address these limitations, we propose Critique-Post-Edit, a robust
reinforcement learning framework that enables more faithful and controllable
personalization. Our framework integrates two key components: (1) a
Personalized Generative Reward Model (GRM) that provides multi-dimensional
scores and textual critiques to resist reward hacking, and (2) a
Critique-Post-Edit mechanism where the policy model revises its own outputs
based on these critiques for more targeted and efficient learning. Under a
rigorous length-controlled evaluation, our method substantially outperforms
standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an
average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses
the performance of GPT-4.1. These results demonstrate a practical path to
faithful, efficient, and controllable personalization.
68f83e8d7669bcaeecce0ce8
Chenghao Zhu
68f83e8d7669bcaeecce0ce9
Meiling Tao
68f83e8d7669bcaeecce0cea
Tiannan Wang
68f83e8d7669bcaeecce0ceb
Dongyi Ding
68f83e8d7669bcaeecce0cec
Yuchen Eleanor Jiang
68f83e8d7669bcaeecce0ced
Wangchunshu Zhou
Towards Faithful and Controllable Personalization via Critique-Post-Edit
  Reinforcement Learning
Faithfully personalizing large language models (LLMs) to align with
individual user preferences is a critical but challenging task. While
supervised fine-tuning (SFT) quickly reaches a performance plateau, standard
reinforcement learning from human feedback (RLHF) also struggles with the
nuances of personalization. Scalar-based reward models are prone to reward
hacking which leads to verbose and superficially personalized responses. To
address these limitations, we propose Critique-Post-Edit, a robust
reinforcement learning framework that enables more faithful and controllable
personalization. Our framework integrates two key components: (1) a
Personalized Generative Reward Model (GRM) that provides multi-dimensional
scores and textual critiques to resist reward hacking, and (2) a
Critique-Post-Edit mechanism where the policy model revises its own outputs
based on these critiques for more targeted and efficient learning. Under a
rigorous length-controlled evaluation, our method substantially outperforms
standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an
average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses
the performance of GPT-4.1. These results demonstrate a practical path to
faithful, efficient, and controllable personalization.
A Critique-Post-Edit framework enhances personalization of large language models by integrating a multi-dimensional reward model and a self-revision mechanism, outperforming standard methods.
supervised fine-tuning reinforcement learning from human feedback reward hacking Personalized Generative Reward Model Critique-Post-Edit policy model length-controlled evaluation PPO personalization benchmarks Qwen2.5-7B Qwen2.5-14B GPT-4.1