Reward-Robust RLHF in LLMs
As Large Language Models (LLMs) continue to progress toward more advanced
forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is
increasingly seen as a key pathway toward achieving Artificial General
Intelligence (AGI). However, the reliance on reward-model-based (RM-based)
alignment methods introduces significant challenges due to the inherent
instability and imperfections of Reward Models (RMs), which can lead to
critical issues such as reward hacking and misalignment with human intentions.
In this paper, we introduce a reward-robust RLHF framework aimed at addressing
these fundamental challenges, paving the way for more reliable and resilient
learning in LLMs. Our approach introduces a novel optimization objective that
carefully balances performance and robustness by incorporating Bayesian Reward
Model Ensembles (BRME) to model the uncertainty set of reward functions. This
allows the framework to integrate both nominal performance and minimum reward
signals, ensuring more stable learning even with imperfect reward models.
Empirical results demonstrate that our framework consistently outperforms
traditional RLHF across diverse benchmarks, showing improved accuracy and
long-term stability. We also provide a theoretical analysis, demonstrating that
reward-robust RLHF approaches the stability of constant reward settings, which
proves to be effective in a stochastic-case analysis. Together, these
contributions highlight the framework potential to enhance both the performance
and stability of LLM alignment with RLHF.
66f395d0685ee4f1ba10ff9d
Yuzi Yan
66f395d0685ee4f1ba10ff9e
Xingzhou Lou
66f395d0685ee4f1ba10ff9f
Jialian Li
66f395d0685ee4f1ba10ffa0
Yiping Zhang
66f395d0685ee4f1ba10ffa1
Jian Xie
66f395d0685ee4f1ba10ffa2
Chao Yu
claimed_verified
2024-10-28T11:27:07.743Z
64ba0f8d842aa47891cb972b
/avatars/3bcacd9b778a146e88e20887b0b00720.svg
Chao Yu
user
zoeyuchao
66f395d0685ee4f1ba10ffa3
Yu Wang
66f395d0685ee4f1ba10ffa4
Dong Yan
66f395d0685ee4f1ba10ffa5
Yuan Shen
Reward-Robust RLHF in LLMs
As Large Language Models (LLMs) continue to progress toward more advanced
forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is
increasingly seen as a key pathway toward achieving Artificial General
Intelligence (AGI). However, the reliance on reward-model-based (RM-based)
alignment methods introduces significant challenges due to the inherent
instability and imperfections of Reward Models (RMs), which can lead to
critical issues such as reward hacking and misalignment with human intentions.
In this paper, we introduce a reward-robust RLHF framework aimed at addressing
these fundamental challenges, paving the way for more reliable and resilient
learning in LLMs. Our approach introduces a novel optimization objective that
carefully balances performance and robustness by incorporating Bayesian Reward
Model Ensembles (BRME) to model the uncertainty set of reward functions. This
allows the framework to integrate both nominal performance and minimum reward
signals, ensuring more stable learning even with imperfect reward models.
Empirical results demonstrate that our framework consistently outperforms
traditional RLHF across diverse benchmarks, showing improved accuracy and
long-term stability. We also provide a theoretical analysis, demonstrating that
reward-robust RLHF approaches the stability of constant reward settings, which
proves to be effective in a stochastic-case analysis. Together, these
contributions highlight the framework potential to enhance both the performance
and stability of LLM alignment with RLHF.
A reward-robust RLHF framework using Bayesian Reward Model Ensembles enhances the stability and performance of LLM alignment with RLHF.
Large Language Models RLHF Artificial General Intelligence Reinforcement Learning from Human Feedback Reward Models reward hacking misalignment Bayesian Reward Model Ensembles BRME