OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human
Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
significantly improve the alignment of human-AI values and further raise the
upper bound of AI capabilities, particularly in reasoning-intensive,
long-context Chain-of-Thought (long-CoT) tasks. However, existing RLHF (or
RLVR) frameworks commonly face challenges such as inference bottlenecks and
complexity barriers, restricting their accessibility for newcomers. To bridge
this gap, we introduce OpenRLHF, a user-friendly, scalable, and easy-to-learn
open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace
Transformers, featuring a simplified design, clear code structure, and
comprehensive documentation to facilitate entry for researchers and
practitioners. Experimental results show that OpenRLHF achieves superior
training efficiency with speedups ranging from 1.22x to 1.68x across different
model sizes compared to state-of-the-art frameworks, while requiring
significantly fewer lines of code for implementation. OpenRLHF is publicly
available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted
by leading institutions to accelerate RLHF research and learning.
664c0b558e6575f45e13d2ad
Jian Hu
extracted_confirmed
2025-06-05T00:20:51.693Z
63f6c04ac96958470d1e9043
/avatars/da46cdd9e21498e120ca91b67bfbfb5e.svg
Jian Hu
user
chuyi777
664c0b558e6575f45e13d2ae
Xibin Wu
admin_assigned
2024-05-21T08:13:06.288Z
64f5a6b46e335217131d7d99
/avatars/c4a15eadc5e612947a314a4a66a51c75.svg
Xibin Wu
user
wuxibin
664c0b558e6575f45e13d2af
Weixun Wang
claimed_verified
2024-05-21T08:59:18.059Z
6263541963f73be3d2f91bdd
/avatars/57474711c7e7768169509eb70e47176e.svg
weixun
user
weixun
664c0b558e6575f45e13d2b0
Xianyu
claimed_verified
2024-05-21T14:15:45.678Z
6054383110aba34e3b6a2f02
/avatars/63ff3a2ce469da0f59e9ef501250a136.svg
Xianyu
user
catqaq
664c0b558e6575f45e13d2b1
Dehao Zhang
admin_assigned
2024-05-21T08:14:28.286Z
634d0eeffc4873ed0d00406a
https://cdn-avatars.huggingface.co/v1/production/uploads/1665994473058-noauth.jpeg
ZhangDehao
user
ispoon
664c0b558e6575f45e13d2b2
Yu Cao
claimed_verified
2024-05-27T07:20:12.327Z
64618ad756c57120a2d1990c
/avatars/1339e57e4d314116a72191d2d2d88f2c.svg
Tony Cao
user
Tony991
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human
Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
significantly improve the alignment of human-AI values and further raise the
upper bound of AI capabilities, particularly in reasoning-intensive,
long-context Chain-of-Thought (long-CoT) tasks. However, existing RLHF (or
RLVR) frameworks commonly face challenges such as inference bottlenecks and
complexity barriers, restricting their accessibility for newcomers. To bridge
this gap, we introduce OpenRLHF, a user-friendly, scalable, and easy-to-learn
open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace
Transformers, featuring a simplified design, clear code structure, and
comprehensive documentation to facilitate entry for researchers and
practitioners. Experimental results show that OpenRLHF achieves superior
training efficiency with speedups ranging from 1.22x to 1.68x across different
model sizes compared to state-of-the-art frameworks, while requiring
significantly fewer lines of code for implementation. OpenRLHF is publicly
available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted
by leading institutions to accelerate RLHF research and learning.
OpenRLHF is an open-source RLHF framework that enhances training efficiency and accessibility for researchers and practitioners.
Reinforcement Learning from Human Feedback RLHF Reinforcement Learning with Verifiable Rewards RLVR Chain-of-Thought long-CoT Ray vLLM DeepSpeed HuggingFace Transformers