Multiplayer Nash Preference Optimization
Reinforcement learning from human feedback (RLHF) has emerged as the standard
paradigm for aligning large language models (LLMs) with human preferences.
However, reward-based methods built on the Bradley-Terry assumption struggle to
capture the non-transitive and heterogeneous nature of real-world preferences.
To address this, recent studies have reframed alignment as a two-player Nash
game, giving rise to Nash learning from human feedback (NLHF). While this
perspective has inspired algorithms such as INPO, ONPO, and EGPO with strong
theoretical and empirical guarantees, they remain fundamentally restricted to
two-player interactions, creating a single-opponent bias that fails to capture
the full complexity of realistic preference structures. In this work, we
introduce Multiplayer Nash Preference Optimization (MNPO), a novel framework
that generalizes NLHF to the multiplayer regime. It formulates alignment as an
n-player game, where each policy competes against a population of opponents
while being regularized toward a reference model. Our framework establishes
well-defined Nash equilibria in multiplayer settings and extends the concept of
duality gap to quantify approximation quality. We demonstrate that MNPO
inherits the equilibrium guarantees of two-player methods while enabling richer
competitive dynamics and improved coverage of diverse preference structures.
Through comprehensive empirical evaluation, we show that MNPO consistently
outperforms existing NLHF baselines on instruction-following benchmarks,
achieving superior alignment quality under heterogeneous annotator conditions
and mixed-policy evaluation scenarios. Together, these results establish MNPO
as a principled and scalable framework for aligning LLMs with complex,
non-transitive human preferences. Code is available at
https://github.com/smiles724/MNPO.
68db42f2d2bf1f4b15ec7323
Fang Wu
claimed_verified
2025-10-02T13:56:41.265Z
675e0d5cdd3e9eeed6954f5a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7oMEoBmaFiCR9K2q9Z_7q.png
Fang Wu
user
fangwu97
68db42f2d2bf1f4b15ec7324
Xu Huang
68db42f2d2bf1f4b15ec7325
Weihao Xuan
claimed_verified
2025-09-30T06:32:41.015Z
65b8909c89eb3dfbe8d26780
https://cdn-avatars.huggingface.co/v1/production/uploads/65b8909c89eb3dfbe8d26780/aJExXsCN9oQOdd9bsQOEO.jpeg
Weihao XUAN
user
weihao1115
68db42f2d2bf1f4b15ec7326
Zhiwei Zhang
68db42f2d2bf1f4b15ec7327
Yijia Xiao
68db42f2d2bf1f4b15ec7328
Guancheng Wan
68db42f2d2bf1f4b15ec7329
Xiaomin Li
68db42f2d2bf1f4b15ec732a
Bing Hu
68db42f2d2bf1f4b15ec732b
Peng Xia
claimed_verified
2025-10-07T12:29:22.751Z
643e9ee6f6bb3c31a26e7bc4
/avatars/acfaa7d6a23dada24c86b954c3be116a.svg
Peng Xia
user
richardxp888
68db42f2d2bf1f4b15ec732c
Jure Leskovec
68db42f2d2bf1f4b15ec732d
Yejin Choi
Multiplayer Nash Preference Optimization
Reinforcement learning from human feedback (RLHF) has emerged as the standard
paradigm for aligning large language models (LLMs) with human preferences.
However, reward-based methods built on the Bradley-Terry assumption struggle to
capture the non-transitive and heterogeneous nature of real-world preferences.
To address this, recent studies have reframed alignment as a two-player Nash
game, giving rise to Nash learning from human feedback (NLHF). While this
perspective has inspired algorithms such as INPO, ONPO, and EGPO with strong
theoretical and empirical guarantees, they remain fundamentally restricted to
two-player interactions, creating a single-opponent bias that fails to capture
the full complexity of realistic preference structures. In this work, we
introduce Multiplayer Nash Preference Optimization (MNPO), a novel framework
that generalizes NLHF to the multiplayer regime. It formulates alignment as an
n-player game, where each policy competes against a population of opponents
while being regularized toward a reference model. Our framework establishes
well-defined Nash equilibria in multiplayer settings and extends the concept of
duality gap to quantify approximation quality. We demonstrate that MNPO
inherits the equilibrium guarantees of two-player methods while enabling richer
competitive dynamics and improved coverage of diverse preference structures.
Through comprehensive empirical evaluation, we show that MNPO consistently
outperforms existing NLHF baselines on instruction-following benchmarks,
achieving superior alignment quality under heterogeneous annotator conditions
and mixed-policy evaluation scenarios. Together, these results establish MNPO
as a principled and scalable framework for aligning LLMs with complex,
non-transitive human preferences. Code is available at
https://github.com/smiles724/MNPO.
Multiplayer Nash Preference Optimization (MNPO) extends Nash learning from human feedback to handle complex, non-transitive human preferences by formulating alignment as an n-player game.
Reinforcement learning from human feedback RLHF large language models LLMs Bradley-Terry assumption Nash learning from human feedback NLHF INPO ONPO EGPO Multiplayer Nash Preference Optimization MNPO Nash equilibria duality gap instruction-following benchmarks heterogeneous annotator conditions mixed-policy evaluation