RLHF Workflow: From Reward Modeling to Online RLHF
We present the workflow of Online Iterative Reinforcement Learning from Human
Feedback (RLHF) in this technical report, which is widely reported to
outperform its offline counterpart by a large margin in the recent large
language model (LLM) literature. However, existing open-source RLHF projects
are still largely confined to the offline learning setting. In this technical
report, we aim to fill in this gap and provide a detailed recipe that is easy
to reproduce for online iterative RLHF. In particular, since online human
feedback is usually infeasible for open-source communities with limited
resources, we start by constructing preference models using a diverse set of
open-source datasets and use the constructed proxy preference model to
approximate human feedback. Then, we discuss the theoretical insights and
algorithmic principles behind online iterative RLHF, followed by a detailed
practical implementation. Our trained LLM, SFR-Iterative-DPO-LLaMA-3-8B-R,
achieves impressive performance on LLM chatbot benchmarks, including
AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks
such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning
(SFT) and iterative RLHF can obtain state-of-the-art performance with fully
open-source datasets. Further, we have made our models, curated datasets, and
comprehensive step-by-step code guidebooks publicly available. Please refer to
https://github.com/RLHFlow/RLHF-Reward-Modeling and
https://github.com/RLHFlow/Online-RLHF for more detailed information.
6642e6abe4a7476619c240f1
Hanze Dong
admin_assigned
2024-05-14T07:12:07.320Z
63a3ff69f91ad3ea5703841d
/avatars/69227c4bce01d33747c1377b6f9672db.svg
Hanze Dong
user
hendrydong
6642e6abe4a7476619c240f2
Wei Xiong
claimed_verified
2024-05-14T07:53:00.588Z
643e59806db6ba8c5ee123f3
/avatars/4052f2a250107f43b3634c3ee3cc30a1.svg
Wei Xiong
user
weqweasdas
6642e6abe4a7476619c240f3
Bo Pang
admin_assigned
2024-05-14T07:12:38.677Z
63e08acbf351dc0745749d56
/avatars/8e2d5ce9db5bd8008ac2ad80f6025553.svg
Bo Pang
user
bpucla
6642e6abe4a7476619c240f4
Haoxiang Wang
admin_assigned
2024-05-14T07:13:11.114Z
638fb8cf2380ffd99caf8c2a
https://cdn-avatars.huggingface.co/v1/production/uploads/638fb8cf2380ffd99caf8c2a/7juUomHJ4gH1HIJj43qE6.jpeg
Haoxiang Wang
user
Haoxiang-Wang
6642e6abe4a7476619c240f5
Han Zhao
6642e6abe4a7476619c240f6
Yingbo Zhou
admin_assigned
2024-05-14T07:13:32.770Z
649bc93758d8b19de0c7785f
/avatars/3ed9473aee23d99f4ee949d3705089ea.svg
Yingbo Zhou
user
yingbozhou
6642e6abe4a7476619c240f7
Nan Jiang
admin_assigned
2024-05-14T07:14:02.032Z
64b8922ca1827cc8d04ae919
/avatars/0aaa83e3d09a82434e1d6af724aaa485.svg
Nan Jiang
user
nanjiang
6642e6abe4a7476619c240f8
Doyen Sahoo
admin_assigned
2024-05-14T07:14:08.581Z
65f84fd980481173afd91233
/avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
Doyen
user
doyensahoo
6642e6abe4a7476619c240f9
Caiming Xiong
admin_assigned
2024-05-14T07:14:15.835Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
6642e6abe4a7476619c240fa
Tong Zhang
RLHF Workflow: From Reward Modeling to Online RLHF
We present the workflow of Online Iterative Reinforcement Learning from Human
Feedback (RLHF) in this technical report, which is widely reported to
outperform its offline counterpart by a large margin in the recent large
language model (LLM) literature. However, existing open-source RLHF projects
are still largely confined to the offline learning setting. In this technical
report, we aim to fill in this gap and provide a detailed recipe that is easy
to reproduce for online iterative RLHF. In particular, since online human
feedback is usually infeasible for open-source communities with limited
resources, we start by constructing preference models using a diverse set of
open-source datasets and use the constructed proxy preference model to
approximate human feedback. Then, we discuss the theoretical insights and
algorithmic principles behind online iterative RLHF, followed by a detailed
practical implementation. Our trained LLM, SFR-Iterative-DPO-LLaMA-3-8B-R,
achieves impressive performance on LLM chatbot benchmarks, including
AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks
such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning
(SFT) and iterative RLHF can obtain state-of-the-art performance with fully
open-source datasets. Further, we have made our models, curated datasets, and
comprehensive step-by-step code guidebooks publicly available. Please refer to
https://github.com/RLHFlow/RLHF-Reward-Modeling and
https://github.com/RLHFlow/Online-RLHF for more detailed information.
Online iterative reinforcement learning from human feedback achieves state-of-the-art performance in large language models using open-source datasets and proxy preference models.
RLHF Online Iterative RLHF preference models proxy preference model SFR-Iterative-DPO-LLaMA-3-8B-R LLM chatbot benchmarks AlpacaEval-2 Arena-Hard MT-Bench HumanEval TruthfulQA supervised fine-tuning SFT