Iterative Nash Policy Optimization: Aligning LLMs with General
  Preferences via No-Regret Learning
Reinforcement Learning with Human Feedback (RLHF) has achieved great success
in aligning large language models (LLMs) with human preferences. Prevalent RLHF
approaches are reward-based, following the Bradley-Terry (BT) model assumption,
which may not fully capture the complexity of human preferences. In this paper,
we explore RLHF under a general preference framework and approach it from a
game-theoretic perspective. Specifically, we formulate the problem as a
two-player game and propose a novel algorithm, iterative Nash policy
optimization (INPO). The key idea is to let the policy play against itself via
no-regret learning, thereby approximating the Nash policy. Unlike previous
methods, INPO bypasses the need for estimating the expected win rate for
individual responses, which typically incurs high computational or annotation
costs. Instead, we introduce a new loss objective that is directly minimized
over a preference dataset. We provide theoretical analysis for our approach and
demonstrate its effectiveness through experiments on various representative
benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 41.5%
length-controlled win rate on AlpacaEval 2.0 and a 38.3% win rate on
Arena-Hard, showing substantial improvement over the state-of-the-art iterative
algorithm [Dong et al., 2024] under the BT model assumption. Additionally, our
ablation study highlights the benefits of incorporating KL regularization for
response length control.
668352f81d6b7ca699ee42dd
Yuheng Zhang
claimed_verified
2024-07-02T07:39:09.289Z
6645466c8899f25c56390ee0
/avatars/4201db32587b67ffa9ddbf08385008f6.svg
Yuheng Zhang
user
MatouK98
668352f81d6b7ca699ee42de
Dian Yu
claimed_verified
2024-07-03T07:41:14.825Z
62d58fd53bf5e059f7cc3245
/avatars/7a4f3ee4a37245f67efd26749d66a706.svg
Dian Yu
user
yudian
668352f81d6b7ca699ee42df
Baolin Peng
668352f81d6b7ca699ee42e0
Linfeng Song
claimed_verified
2025-07-11T08:22:00.287Z
64c94eddcb2f1bf0e7db5a4d
/avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
Linfeng Song
user
freesunshine0316
668352f81d6b7ca699ee42e1
Ye Tian
claimed_verified
2024-10-17T11:29:06.324Z
6414061f996b2e426f23df48
/avatars/db0af49c5f017f7c455a076b8b2212b6.svg
Ye Tian
user
yetian
668352f81d6b7ca699ee42e2
Mingyue Huo
668352f81d6b7ca699ee42e3
Nan Jiang
668352f81d6b7ca699ee42e4
Haitao Mi
668352f81d6b7ca699ee42e5
Dong Yu
Iterative Nash Policy Optimization: Aligning LLMs with General
  Preferences via No-Regret Learning
Reinforcement Learning with Human Feedback (RLHF) has achieved great success
in aligning large language models (LLMs) with human preferences. Prevalent RLHF
approaches are reward-based, following the Bradley-Terry (BT) model assumption,
which may not fully capture the complexity of human preferences. In this paper,
we explore RLHF under a general preference framework and approach it from a
game-theoretic perspective. Specifically, we formulate the problem as a
two-player game and propose a novel algorithm, iterative Nash policy
optimization (INPO). The key idea is to let the policy play against itself via
no-regret learning, thereby approximating the Nash policy. Unlike previous
methods, INPO bypasses the need for estimating the expected win rate for
individual responses, which typically incurs high computational or annotation
costs. Instead, we introduce a new loss objective that is directly minimized
over a preference dataset. We provide theoretical analysis for our approach and
demonstrate its effectiveness through experiments on various representative
benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 41.5%
length-controlled win rate on AlpacaEval 2.0 and a 38.3% win rate on
Arena-Hard, showing substantial improvement over the state-of-the-art iterative
algorithm [Dong et al., 2024] under the BT model assumption. Additionally, our
ablation study highlights the benefits of incorporating KL regularization for
response length control.
A novel algorithm, iterative Nash policy optimization (INPO), is proposed for Reinforcement Learning with Human Feedback (RLHF) under a general preference framework, achieving better performance than existing methods on multiple benchmarks.
Reinforcement Learning with Human Feedback (RLHF) large language models (LLMs) Bradley-Terry (BT) model iterative Nash policy optimization (INPO) no-regret learning Nash policy preference dataset theoretical analysis AlpacaEval 2.0 Arena-Hard SFT model KL regularization