Nash Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) has emerged as the main
paradigm for aligning large language models (LLMs) with human preferences.
Typically, RLHF involves the initial step of learning a reward model from human
feedback, often expressed as preferences between pairs of text generations
produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by
optimizing it to maximize the reward model through a reinforcement learning
algorithm. However, an inherent limitation of current reward models is their
inability to fully represent the richness of human preferences and their
dependency on the sampling distribution.
  In this study, we introduce an alternative pipeline for the fine-tuning of
LLMs using pairwise human feedback. Our approach entails the initial learning
of a preference model, which is conditioned on two inputs given a prompt,
followed by the pursuit of a policy that consistently generates responses
preferred over those generated by any competing policy, thus defining the Nash
equilibrium of this preference model. We term this approach Nash learning from
human feedback (NLHF).
  In the context of a tabular policy representation, we present a novel
algorithmic solution, Nash-MD, founded on the principles of mirror descent.
This algorithm produces a sequence of policies, with the last iteration
converging to the regularized Nash equilibrium. Additionally, we explore
parametric representations of policies and introduce gradient descent
algorithms for deep-learning architectures. To demonstrate the effectiveness of
our approach, we present experimental results involving the fine-tuning of a
LLM for a text summarization task. We believe NLHF offers a compelling avenue
for preference learning and policy optimization with the potential of advancing
the field of aligning LLMs with human preferences.
656e9a8c7ef5b6b3b85b7d35
Rémi Munos
656e9a8c7ef5b6b3b85b7d36
Michal Valko
claimed_verified
2024-01-18T19:42:37.801Z
651e97156d92456bdf5ace6b
https://cdn-avatars.huggingface.co/v1/production/uploads/651e97156d92456bdf5ace6b/KKfdZGPAcWPdqycp9SulH.jpeg
Michal Valko
user
misovalko
656e9a8c7ef5b6b3b85b7d37
Daniele Calandriello
656e9a8c7ef5b6b3b85b7d38
Mohammad Gheshlaghi Azar
656e9a8c7ef5b6b3b85b7d39
Mark Rowland
656e9a8c7ef5b6b3b85b7d3a
Daniel Guo
656e9a8c7ef5b6b3b85b7d3b
Yunhao Tang
656e9a8c7ef5b6b3b85b7d3c
Matthieu Geist
claimed_verified
2025-05-19T13:09:53.085Z
67bef153eaca0bef78ed5478
/avatars/ab838c541bd170986370a99670981c2f.svg
Matthieu Geist
user
matthieu-geist
656e9a8c7ef5b6b3b85b7d3d
Thomas Mésnard
656e9a8c7ef5b6b3b85b7d3e
Andrea Michi
656e9a8c7ef5b6b3b85b7d3f
Marco Selvi
656e9a8c7ef5b6b3b85b7d40
Sertan Girgin
656e9a8c7ef5b6b3b85b7d41
Nikola Momchev
656e9a8c7ef5b6b3b85b7d42
Olivier Bachem
656e9a8c7ef5b6b3b85b7d43
Daniel J. Mankowitz
656e9a8c7ef5b6b3b85b7d44
Doina Precup
656e9a8c7ef5b6b3b85b7d45
Bilal Piot
Nash Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) has emerged as the main
paradigm for aligning large language models (LLMs) with human preferences.
Typically, RLHF involves the initial step of learning a reward model from human
feedback, often expressed as preferences between pairs of text generations
produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by
optimizing it to maximize the reward model through a reinforcement learning
algorithm. However, an inherent limitation of current reward models is their
inability to fully represent the richness of human preferences and their
dependency on the sampling distribution.
  In this study, we introduce an alternative pipeline for the fine-tuning of
LLMs using pairwise human feedback. Our approach entails the initial learning
of a preference model, which is conditioned on two inputs given a prompt,
followed by the pursuit of a policy that consistently generates responses
preferred over those generated by any competing policy, thus defining the Nash
equilibrium of this preference model. We term this approach Nash learning from
human feedback (NLHF).
  In the context of a tabular policy representation, we present a novel
algorithmic solution, Nash-MD, founded on the principles of mirror descent.
This algorithm produces a sequence of policies, with the last iteration
converging to the regularized Nash equilibrium. Additionally, we explore
parametric representations of policies and introduce gradient descent
algorithms for deep-learning architectures. To demonstrate the effectiveness of
our approach, we present experimental results involving the fine-tuning of a
LLM for a text summarization task. We believe NLHF offers a compelling avenue
for preference learning and policy optimization with the potential of advancing
the field of aligning LLMs with human preferences.
NLHF uses a preference model and mirror descent to fine-tune LLMs for text summarization, advancing alignment with human preferences.
reinforcement learning from human feedback (RLHF) large language models (LLMs) reward model pairwise human feedback preference model Nash equilibrium Nash-MD mirror descent parametric representations gradient descent deep-learning architectures text summarization