RLBFF: Binary Flexible Feedback to bridge between Human Feedback &
  Verifiable Rewards
Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning
with Verifiable Rewards (RLVR) are the main RL paradigms used in LLM
post-training, each offering distinct advantages. However, RLHF struggles with
interpretability and reward hacking because it relies on human judgments that
usually lack explicit criteria, whereas RLVR is limited in scope by its focus
on correctness-based verifiers. We propose Reinforcement Learning with Binary
Flexible Feedback (RLBFF), which combines the versatility of human-driven
preferences with the precision of rule-based verification, enabling reward
models to capture nuanced aspects of response quality beyond mere correctness.
RLBFF extracts principles that can be answered in a binary fashion (e.g.
accuracy of information: yes, or code readability: no) from natural language
feedback. Such principles can then be used to ground Reward Model training as
an entailment task (response satisfies or does not satisfy an arbitrary
principle). We show that Reward Models trained in this manner can outperform
Bradley-Terry models when matched for data and achieve top performance on
RM-Bench (86.2%) and JudgeBench (81.4%, #1 on leaderboard as of September 24,
2025). Additionally, users can specify principles of interest at inference time
to customize the focus of our reward models, in contrast to Bradley-Terry
models. Finally, we present a fully open source recipe (including data) to
align Qwen3-32B using RLBFF and our Reward Model, to match or exceed the
performance of o3-mini and DeepSeek R1 on general alignment benchmarks of
MT-Bench, WildBench, and Arena Hard v2 (at <5% of the inference cost).
68dab42fd2bf1f4b15ec7178
Zhilin Wang
claimed_verified
2025-09-30T06:35:43.581Z
635865c8aff68f72ac039a4e
/avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
Zhilin Wang
user
zhilinw
68dab42fd2bf1f4b15ec7179
Jiaqi Zeng
68dab42fd2bf1f4b15ec717a
Olivier Delalleau
68dab42fd2bf1f4b15ec717b
Ellie Evans
68dab42fd2bf1f4b15ec717c
Daniel Egert
68dab42fd2bf1f4b15ec717d
Hoo-Chang Shin
68dab42fd2bf1f4b15ec717e
Felipe Soares
68dab42fd2bf1f4b15ec717f
Yi Dong
68dab42fd2bf1f4b15ec7180
Oleksii Kuchaiev
RLBFF: Binary Flexible Feedback to bridge between Human Feedback &
  Verifiable Rewards
Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning
with Verifiable Rewards (RLVR) are the main RL paradigms used in LLM
post-training, each offering distinct advantages. However, RLHF struggles with
interpretability and reward hacking because it relies on human judgments that
usually lack explicit criteria, whereas RLVR is limited in scope by its focus
on correctness-based verifiers. We propose Reinforcement Learning with Binary
Flexible Feedback (RLBFF), which combines the versatility of human-driven
preferences with the precision of rule-based verification, enabling reward
models to capture nuanced aspects of response quality beyond mere correctness.
RLBFF extracts principles that can be answered in a binary fashion (e.g.
accuracy of information: yes, or code readability: no) from natural language
feedback. Such principles can then be used to ground Reward Model training as
an entailment task (response satisfies or does not satisfy an arbitrary
principle). We show that Reward Models trained in this manner can outperform
Bradley-Terry models when matched for data and achieve top performance on
RM-Bench (86.2%) and JudgeBench (81.4%, #1 on leaderboard as of September 24,
2025). Additionally, users can specify principles of interest at inference time
to customize the focus of our reward models, in contrast to Bradley-Terry
models. Finally, we present a fully open source recipe (including data) to
align Qwen3-32B using RLBFF and our Reward Model, to match or exceed the
performance of o3-mini and DeepSeek R1 on general alignment benchmarks of
MT-Bench, WildBench, and Arena Hard v2 (at <5% of the inference cost).
RLBFF combines human feedback and rule-based verification to improve reward models, achieving top performance on benchmarks with customizable principles.
Reinforcement Learning with Human Feedback (RLHF) Reinforcement Learning with Verifiable Rewards (RLVR) Reinforcement Learning with Binary Flexible Feedback (RLBFF) reward models entailment task RM-Bench JudgeBench MT-Bench WildBench Arena Hard v2