Open Problems and Fundamental Limitations of Reinforcement Learning from
  Human Feedback
Reinforcement learning from human feedback (RLHF) is a technique for training
AI systems to align with human goals. RLHF has emerged as the central method
used to finetune state-of-the-art large language models (LLMs). Despite this
popularity, there has been relatively little public work systematizing its
flaws. In this paper, we (1) survey open problems and fundamental limitations
of RLHF and related methods; (2) overview techniques to understand, improve,
and complement RLHF in practice; and (3) propose auditing and disclosure
standards to improve societal oversight of RLHF systems. Our work emphasizes
the limitations of RLHF and highlights the importance of a multi-faceted
approach to the development of safer AI systems.
64c74566589826641bcfccd7
Stephen Casper
admin_assigned
2023-07-31T06:21:48.917Z
6466a046326128fd2c6c59c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6466a046326128fd2c6c59c2/kYwPcxupelOBvKFB0y8Me.png
Stephen Casper
user
stecas
64c74566589826641bcfccd8
Xander Davies
admin_assigned
2023-07-31T06:21:57.219Z
62d689078faee0ac953b97c4
/avatars/1c35b888bb36046439ab00a18c609eb1.svg
Alexander Davies
user
xanderdavies
64c74566589826641bcfccd9
Claudia Shi
64c74566589826641bcfccda
Thomas Krendl Gilbert
64c74566589826641bcfccdb
Jérémy Scheurer
64c74566589826641bcfccdc
Javier Rando
admin_assigned
2023-07-31T06:23:27.907Z
6245b1b2d5895bdf34ef8c33
https://cdn-avatars.huggingface.co/v1/production/uploads/6245b1b2d5895bdf34ef8c33/hLmcppWpRS7s2rxMxdWGh.jpeg
Javier Rando
user
javirandor
64c74566589826641bcfccdd
Rachel Freedman
claimed_verified
2023-08-01T07:44:55.055Z
64c7d8173d5a0dfed5a612ce
/avatars/738b6113e35420e3d20f277db8c38416.svg
Rachel Freedman
user
RachelFreedman
64c74566589826641bcfccde
Tomasz Korbak
64c74566589826641bcfccdf
David Lindner
64c74566589826641bcfcce0
Pedro Freire
64c74566589826641bcfcce1
Tony Wang
64c74566589826641bcfcce2
Samuel Marks
64c74566589826641bcfcce3
Charbel-Raphaël Segerie
64c74566589826641bcfcce4
Micah Carroll
claimed_verified
2023-08-01T07:44:51.608Z
625cba98aa13d02f2a8d9029
/avatars/3b557357ee9ae07dd12ba930f3942e17.svg
Carroll
user
Micah
64c74566589826641bcfcce5
Andi Peng
64c74566589826641bcfcce6
Phillip Christoffersen
64c74566589826641bcfcce7
Mehul Damani
claimed_verified
2025-07-29T12:51:57.547Z
649da680e0fff1ed099ae92f
/avatars/de40dc02c2285bfb854037bfd29f7fb9.svg
Mehul Damani
user
mehuldamani
64c74566589826641bcfcce8
Stewart Slocum
admin_assigned
2023-07-31T06:26:55.278Z
64b71864efefc8a73877c2c0
https://cdn-avatars.huggingface.co/v1/production/uploads/64b71864efefc8a73877c2c0/Xdwn0KsSNev-iBrlnXh0o.jpeg
Stewart Slocum
user
stewy33
64c74566589826641bcfcce9
Usman Anwar
64c74566589826641bcfccea
Anand Siththaranjan
64c74566589826641bcfcceb
Max Nadeau
64c74566589826641bcfccec
Eric J. Michaud
claimed_verified
2024-05-24T12:03:37.262Z
6303e59c7b50dd9d0a367375
/avatars/353aaaae6db1c7dc89895927a65fa9b1.svg
Eric Michaud
user
ericjm
64c74566589826641bcfcced
Jacob Pfau
admin_assigned
2023-07-31T06:47:31.412Z
63766f562361581ceb23d34f
/avatars/8563015693f5e0e03c25d39034cae599.svg
Jacob Pfau
user
jacobpfau
64c74566589826641bcfccee
Dmitrii Krasheninnikov
64c74566589826641bcfccef
Xin Chen
64c74566589826641bcfccf0
Lauro Langosco
64c74566589826641bcfccf1
Peter Hase
64c74566589826641bcfccf2
Erdem Bıyık
admin_assigned
2023-07-31T06:49:18.212Z
644387fe9504687e2d82f0e5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/wx1emudpMZ0BzdnqI7Xnl.png
Erdem Bıyık
user
ebiyik
64c74566589826641bcfccf3
Anca Dragan
64c74566589826641bcfccf4
David Krueger
64c74566589826641bcfccf5
Dorsa Sadigh
64c74566589826641bcfccf6
Dylan Hadfield-Menell
admin_assigned
2023-07-31T06:50:02.742Z
64905f22dc0d89dca8f0db13
/avatars/7e30391d6b8914806724afbcd7401a29.svg
Dylan Hadfield-Menell
user
dhadfieldmenell
Open Problems and Fundamental Limitations of Reinforcement Learning from
  Human Feedback
Reinforcement learning from human feedback (RLHF) is a technique for training
AI systems to align with human goals. RLHF has emerged as the central method
used to finetune state-of-the-art large language models (LLMs). Despite this
popularity, there has been relatively little public work systematizing its
flaws. In this paper, we (1) survey open problems and fundamental limitations
of RLHF and related methods; (2) overview techniques to understand, improve,
and complement RLHF in practice; and (3) propose auditing and disclosure
standards to improve societal oversight of RLHF systems. Our work emphasizes
the limitations of RLHF and highlights the importance of a multi-faceted
approach to the development of safer AI systems.
RLHF is analyzed for its limitations and flaws, with suggestions for improvement, complementation, and oversight to enhance safer AI development.
reinforcement learning from human feedback RLHF large language models LLMs auditing disclosure standards