Reinforcement Learning with Rubric Anchors
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a
powerful paradigm for enhancing Large Language Models (LLMs), exemplified by
the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable
signals-such as passing unit tests in code generation or matching correct
answers in mathematical reasoning. While effective, this requirement largely
confines RLVR to domains with automatically checkable outcomes. To overcome
this, we extend the RLVR paradigm to open-ended tasks by integrating
rubric-based rewards, where carefully designed rubrics serve as structured,
model-interpretable criteria for automatic scoring of subjective outputs. We
construct, to our knowledge, the largest rubric reward system to date, with
over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration.
Implementing rubric-based RL is challenging; we tackle these issues with a
clear framework and present an open-sourced Qwen-30B-A3B model with notable
gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended
benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by
+2.4%, while preserving general and reasoning abilities. 2) Our method provides
fine-grained stylistic control, using rubrics as anchors to mitigate the
"AI-like" tone and produce more human-like, expressive responses. We share key
lessons in rubric construction, data selection, and training, and discuss
limitations and future releases.
68a42f43b65388761d0745c1
Zenan Huang
claimed_verified
2025-08-20T08:52:50.689Z
625f8e6f673e5862a8c07f1a
https://cdn-avatars.huggingface.co/v1/production/uploads/1677306231208-625f8e6f673e5862a8c07f1a.jpeg
Bill H
user
lccurious
68a42f43b65388761d0745c2
Yihong Zhuang
claimed_verified
2025-08-19T10:48:36.226Z
673b5f24e863f1d28b402efc
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/19gUgtPEY3-FtY0sNlI_-.png
yihongzhuang
user
utdawn
68a42f43b65388761d0745c3
Guoshan Lu
claimed_verified
2025-08-20T08:52:35.073Z
649ee7ac1f44b2c0301df24e
/avatars/00864ad358cd5e0416c03b9587dfc56e.svg
Guoshan Lu
user
grason-lu
68a42f43b65388761d0745c4
Zeyu Qin
68a42f43b65388761d0745c5
Haokai Xu
68a42f43b65388761d0745c6
Tianyu Zhao
claimed_verified
2025-08-21T09:16:20.862Z
67bfdad6debf43fd3aeb8b81
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vC7cTEsduxptifomE7OM9.png
TIANYU
user
kokogigi
68a42f43b65388761d0745c7
Ru Peng
68a42f43b65388761d0745c8
Jiaqi Hu
68a42f43b65388761d0745c9
Zhanming Shen
68a42f43b65388761d0745ca
Xiaomeng Hu
68a42f43b65388761d0745cb
Xijun Gu
68a42f43b65388761d0745cc
Peiyi Tu
68a42f43b65388761d0745cd
Jiaxin Liu
68a42f43b65388761d0745ce
Wenyu Chen
68a42f43b65388761d0745cf
Yuzhuo Fu
68a42f43b65388761d0745d0
Zhiting Fan
68a42f43b65388761d0745d1
Yanmei Gu
68a42f43b65388761d0745d2
Yuanyuan Wang
68a42f43b65388761d0745d3
Zhengkai Yang
68a42f43b65388761d0745d4
Jianguo Li
68a42f43b65388761d0745d5
Junbo Zhao
Reinforcement Learning with Rubric Anchors
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a
powerful paradigm for enhancing Large Language Models (LLMs), exemplified by
the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable
signals-such as passing unit tests in code generation or matching correct
answers in mathematical reasoning. While effective, this requirement largely
confines RLVR to domains with automatically checkable outcomes. To overcome
this, we extend the RLVR paradigm to open-ended tasks by integrating
rubric-based rewards, where carefully designed rubrics serve as structured,
model-interpretable criteria for automatic scoring of subjective outputs. We
construct, to our knowledge, the largest rubric reward system to date, with
over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration.
Implementing rubric-based RL is challenging; we tackle these issues with a
clear framework and present an open-sourced Qwen-30B-A3B model with notable
gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended
benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by
+2.4%, while preserving general and reasoning abilities. 2) Our method provides
fine-grained stylistic control, using rubrics as anchors to mitigate the
"AI-like" tone and produce more human-like, expressive responses. We share key
lessons in rubric construction, data selection, and training, and discuss
limitations and future releases.
RLVR is extended to open-ended tasks using rubric-based rewards, improving performance on benchmarks and providing stylistic control in LLMs.
Reinforcement Learning from Verifiable Rewards RLVR Large Language Models LLMs rubric-based rewards Qwen-30B-A3B DeepSeek-V3 open-ended benchmarks stylistic control