OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward
  Modeling and LLM Alignment
Reward modeling lies at the core of reinforcement learning from human
feedback (RLHF), yet most existing reward models rely on scalar or pairwise
judgments that fail to capture the multifaceted nature of human preferences.
Recent studies have explored rubrics-as-rewards (RaR) that uses structured
natural language criteria that capture multiple dimensions of response quality.
However, producing rubrics that are both reliable and scalable remains a key
challenge. In this work, we introduce OpenRubrics, a diverse, large-scale
collection of (prompt, rubric) pairs for training rubric-generation and
rubric-based reward models. To elicit discriminative and comprehensive
evaluation signals, we introduce Contrastive Rubric Generation (CRG), which
derives both hard rules (explicit constraints) and principles (implicit
qualities) by contrasting preferred and rejected responses. We further improve
reliability by enforcing preference-label consistency via rejection sampling to
remove noisy rubrics. Across multiple reward-modeling benchmarks, our
rubric-based reward model, Rubric-RM, surpasses strong size-matched baselines
by 6.8%. These gains transfer to policy models on instruction-following and
biomedical benchmarks. Our results show that rubrics provide scalable alignment
signals that narrow the gap between costly human evaluation and automated
reward modeling, enabling a new principle-driven paradigm for LLM alignment.
68e9119d95e8e6771df38ca3
Tianci Liu
claimed_verified
2025-10-17T04:14:27.720Z
64bf811d76a6e2efcceabc00
https://cdn-avatars.huggingface.co/v1/production/uploads/64bf811d76a6e2efcceabc00/0p3zSIVqzoME25Zmfh7SD.png
Tianci Liu
user
lliutianc
68e9119d95e8e6771df38ca4
Ran Xu
claimed_verified
2025-10-13T10:10:00.598Z
6621dd7623518bf2898788da
/avatars/837aca71471a19ea80ae55d619781c5a.svg
Ran Xu
user
ritaranx
68e9119d95e8e6771df38ca5
Tony Yu
68e9119d95e8e6771df38ca6
Ilgee Hong
68e9119d95e8e6771df38ca7
Carl Yang
68e9119d95e8e6771df38ca8
Tuo Zhao
68e9119d95e8e6771df38ca9
Haoyu Wang
claimed_verified
2025-10-11T13:45:38.132Z
641a92bc4182690729c9324b
/avatars/f5d3de7f04fe77d0cfced51b5431c114.svg
haoyu wang
user
haoyuw
OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward
  Modeling and LLM Alignment
Reward modeling lies at the core of reinforcement learning from human
feedback (RLHF), yet most existing reward models rely on scalar or pairwise
judgments that fail to capture the multifaceted nature of human preferences.
Recent studies have explored rubrics-as-rewards (RaR) that uses structured
natural language criteria that capture multiple dimensions of response quality.
However, producing rubrics that are both reliable and scalable remains a key
challenge. In this work, we introduce OpenRubrics, a diverse, large-scale
collection of (prompt, rubric) pairs for training rubric-generation and
rubric-based reward models. To elicit discriminative and comprehensive
evaluation signals, we introduce Contrastive Rubric Generation (CRG), which
derives both hard rules (explicit constraints) and principles (implicit
qualities) by contrasting preferred and rejected responses. We further improve
reliability by enforcing preference-label consistency via rejection sampling to
remove noisy rubrics. Across multiple reward-modeling benchmarks, our
rubric-based reward model, Rubric-RM, surpasses strong size-matched baselines
by 6.8%. These gains transfer to policy models on instruction-following and
biomedical benchmarks. Our results show that rubrics provide scalable alignment
signals that narrow the gap between costly human evaluation and automated
reward modeling, enabling a new principle-driven paradigm for LLM alignment.
Rubric-based reward models using OpenRubrics and Contrastive Rubric Generation improve alignment in reinforcement learning from human feedback by providing scalable and reliable evaluation signals.
reinforcement learning from human feedback reward models rubrics-as-rewards structured natural language criteria Contrastive Rubric Generation hard rules principles preference-label consistency rejection sampling Rubric-RM instruction-following biomedical benchmarks LLM alignment