Robust Reward Modeling via Causal Rubrics
Reward models (RMs) are fundamental to aligning Large Language Models (LLMs)
via human feedback, yet they often suffer from reward hacking. They tend to
latch on to superficial or spurious attributes, such as response length or
formatting, mistaking these cues learned from correlations in training data for
the true causal drivers of quality (e.g., factuality, relevance). This occurs
because standard training objectives struggle to disentangle these factors,
leading to brittle RMs and misaligned policies. We introduce Crome (Causally
Robust Reward Modeling), a novel framework grounded in an explicit causal model
designed to mitigate reward hacking. Crome employs the following synthetic
targeted augmentations during training: (1) Causal Augmentations, which are
pairs that differ along specific causal attributes, to enforce sensitivity
along each causal attribute individually, and (2) Neutral Augmentations, which
are tie-label pairs varying primarily in spurious attributes, to enforce
invariance along spurious attributes. Notably, our augmentations are produced
without any knowledge of spurious factors, via answer interventions only along
causal rubrics, that are identified by querying an oracle LLM. Empirically,
Crome significantly outperforms standard baselines on RewardBench, improving
average accuracy by up to 5.4% and achieving gains of up to 13.2% and 7.2% in
specific categories. The robustness of Crome is further testified by the
consistent gains obtained in a Best-of-N inference setting across increasing N,
across various benchmarks, including the popular RewardBench (covering chat,
chat-hard, safety, and reasoning tasks), the safety-focused WildGuardTest, and
the reasoning-specific GSM8k.
685a752f0e4ad7e21975870a
Pragya Srivastava
admin_assigned
2025-06-24T13:33:25.360Z
639ccab166106be1436e1640
/avatars/1e3806e18ac427be20e93e5400f153d4.svg
Pragya Srivastava
user
pragsri8
685a752f0e4ad7e21975870b
Harman Singh
claimed_verified
2025-06-25T20:59:30.863Z
631c4048e65cd7f686d2ad4f
/avatars/49902bcd18da8079bacf312edc386982.svg
Harman Singh
user
harman
685a752f0e4ad7e21975870c
Rahul Madhavan
685a752f0e4ad7e21975870d
Gandharv Patil
admin_assigned
2025-06-24T13:33:41.718Z
65fbbfa4dae495e776dfaf50
/avatars/75630742e1c3ae93f1ba9ee900e22cf0.svg
Gandharv Patil
user
gp02-mcgill
685a752f0e4ad7e21975870e
Sravanti Addepalli
admin_assigned
2025-06-24T13:33:47.759Z
654355390d2e27122a64d7db
/avatars/38f93647d1f98af387762781fbdff5de.svg
Sravanti Addepalli
user
a-sravanti
685a752f0e4ad7e21975870f
Arun Suggala
685a752f0e4ad7e219758710
Rengarajan Aravamudhan
685a752f0e4ad7e219758711
Soumya Sharma
685a752f0e4ad7e219758712
Anirban Laha
admin_assigned
2025-06-24T13:34:03.892Z
660bc971a276be904ac1d5fc
/avatars/543a5d18710ba6260d5ea635f99f37ea.svg
Anirban Laha
user
anirbanl
685a752f0e4ad7e219758713
Aravindan Raghuveer
admin_assigned
2025-06-24T13:34:13.363Z
6208f528ace0f815845c6de0
/avatars/6d31e68edb7fc1680e12e0939470a6e5.svg
Aravindan R
user
Aravindan
685a752f0e4ad7e219758714
Karthikeyan Shanmugam
685a752f0e4ad7e219758715
Doina Precup
Robust Reward Modeling via Causal Rubrics
Reward models (RMs) are fundamental to aligning Large Language Models (LLMs)
via human feedback, yet they often suffer from reward hacking. They tend to
latch on to superficial or spurious attributes, such as response length or
formatting, mistaking these cues learned from correlations in training data for
the true causal drivers of quality (e.g., factuality, relevance). This occurs
because standard training objectives struggle to disentangle these factors,
leading to brittle RMs and misaligned policies. We introduce Crome (Causally
Robust Reward Modeling), a novel framework grounded in an explicit causal model
designed to mitigate reward hacking. Crome employs the following synthetic
targeted augmentations during training: (1) Causal Augmentations, which are
pairs that differ along specific causal attributes, to enforce sensitivity
along each causal attribute individually, and (2) Neutral Augmentations, which
are tie-label pairs varying primarily in spurious attributes, to enforce
invariance along spurious attributes. Notably, our augmentations are produced
without any knowledge of spurious factors, via answer interventions only along
causal rubrics, that are identified by querying an oracle LLM. Empirically,
Crome significantly outperforms standard baselines on RewardBench, improving
average accuracy by up to 5.4% and achieving gains of up to 13.2% and 7.2% in
specific categories. The robustness of Crome is further testified by the
consistent gains obtained in a Best-of-N inference setting across increasing N,
across various benchmarks, including the popular RewardBench (covering chat,
chat-hard, safety, and reasoning tasks), the safety-focused WildGuardTest, and
the reasoning-specific GSM8k.
Crome, a novel reward modeling framework using causal and neutral augmentations, significantly improves the robustness and accuracy of reward models against reward hacking.
Reward models Large Language Models reward hacking causal model causal augmentations neutral augmentations answer interventions oracle LLM RewardBench WildGuardTest GSM8k Best-of-N inference