Chasing the Tail: Effective Rubric-based Reward Modeling for Large
  Language Model Post-Training
Reinforcement fine-tuning (RFT) often suffers from reward
over-optimization, where a policy model hacks the reward signals to achieve
high scores while producing low-quality outputs. Our theoretical analysis shows
that the key lies in reward misspecification at the high-reward tail: the
inability to reliably distinguish Excellent responses from merely Great ones.
This motivate us to focus on the high-reward region. However, such tail
examples are scarce under the base LLM. While off-policy exemplars (e.g. from
stronger models or rewrites) are easier to obtain, naively training on them
yields a misspecified reward for the policy we aim to align. To address this,
we study rubric-based rewards. By design, rubrics can leverage off-policy
examples while remaining insensitive to their artifacts. To elicit rubrics that
capture the high-reward tail, we highlight the importance of distinguishing
among great and diverse responses, and introduce a workflow to implement this
idea. We empirically demonstrate that rubric-based rewards substantially
mitigate reward over-optimization and deliver effective LLM post-training
improvements. Our code can be accessed at
https://github.com/Jun-Kai-Zhang/rubrics.git .
68da0e7e0177a6054b013b9b
Junkai Zhang
68da0e7e0177a6054b013b9c
Zihao Wang
68da0e7e0177a6054b013b9d
Lin Gui
68da0e7e0177a6054b013b9e
Swarnashree Mysore Sathyendra
claimed_verified
2025-10-10T09:56:48.011Z
6483a713310dbc328305206c
/avatars/e04270cbf38c822362c171f80f1f67d3.svg
Swarnashree
user
mysoress
68da0e7e0177a6054b013b9f
Jaehwan Jeong
68da0e7e0177a6054b013ba0
Victor Veitch
68da0e7e0177a6054b013ba1
Wei Wang
68da0e7e0177a6054b013ba2
Yunzhong He
68da0e7e0177a6054b013ba3
Bing Liu
68da0e7e0177a6054b013ba4
Lifeng Jin
Chasing the Tail: Effective Rubric-based Reward Modeling for Large
  Language Model Post-Training
Reinforcement fine-tuning (RFT) often suffers from reward
over-optimization, where a policy model hacks the reward signals to achieve
high scores while producing low-quality outputs. Our theoretical analysis shows
that the key lies in reward misspecification at the high-reward tail: the
inability to reliably distinguish Excellent responses from merely Great ones.
This motivate us to focus on the high-reward region. However, such tail
examples are scarce under the base LLM. While off-policy exemplars (e.g. from
stronger models or rewrites) are easier to obtain, naively training on them
yields a misspecified reward for the policy we aim to align. To address this,
we study rubric-based rewards. By design, rubrics can leverage off-policy
examples while remaining insensitive to their artifacts. To elicit rubrics that
capture the high-reward tail, we highlight the importance of distinguishing
among great and diverse responses, and introduce a workflow to implement this
idea. We empirically demonstrate that rubric-based rewards substantially
mitigate reward over-optimization and deliver effective LLM post-training
improvements. Our code can be accessed at
https://github.com/Jun-Kai-Zhang/rubrics.git .
Rubric-based rewards mitigate reward over-optimization in reinforcement fine-tuning by leveraging off-policy examples while maintaining reward reliability.
reinforcement fine-tuning reward over-optimization policy model reward misspecification high-reward region base LLM off-policy exemplars rubric-based rewards reward alignment LLM post-training improvements