Learning Explainable Dense Reward Shapes via Bayesian Optimization
Current reinforcement learning from human feedback (RLHF) pipelines for large
language model (LLM) alignment typically assign scalar rewards to sequences,
using the final token as a surrogate indicator for the quality of the entire
sequence. However, this leads to sparse feedback and suboptimal token-level
credit assignment. In this work, we frame reward shaping as an optimization
problem focused on token-level credit assignment. We propose a reward-shaping
function leveraging explainability methods such as SHAP and LIME to estimate
per-token rewards from the reward model. To learn parameters of this shaping
function, we employ a bilevel optimization framework that integrates Bayesian
Optimization and policy training to handle noise from the token reward
estimates. Our experiments show that achieving a better balance of token-level
reward attribution leads to performance improvements over baselines on
downstream tasks and finds an optimal policy faster during training.
Furthermore, we show theoretically that explainability methods that are feature
additive attribution functions maintain the optimal policy as the original
reward.
68127237228929e037039e84
Ryan Koo
68127237228929e037039e85
Ian Yang
extracted_pending
2025-04-30T18:55:52.538Z
649339184005b1c3c021529a
/avatars/67391550642d2a476d388b4898134b8c.svg
Ian Yang
user
iyang30
68127237228929e037039e86
Vipul Raheja
68127237228929e037039e87
Mingyi Hong
68127237228929e037039e88
Kwang-Sung Jun
68127237228929e037039e89
Dongyeop Kang
Learning Explainable Dense Reward Shapes via Bayesian Optimization
Current reinforcement learning from human feedback (RLHF) pipelines for large
language model (LLM) alignment typically assign scalar rewards to sequences,
using the final token as a surrogate indicator for the quality of the entire
sequence. However, this leads to sparse feedback and suboptimal token-level
credit assignment. In this work, we frame reward shaping as an optimization
problem focused on token-level credit assignment. We propose a reward-shaping
function leveraging explainability methods such as SHAP and LIME to estimate
per-token rewards from the reward model. To learn parameters of this shaping
function, we employ a bilevel optimization framework that integrates Bayesian
Optimization and policy training to handle noise from the token reward
estimates. Our experiments show that achieving a better balance of token-level
reward attribution leads to performance improvements over baselines on
downstream tasks and finds an optimal policy faster during training.
Furthermore, we show theoretically that explainability methods that are feature
additive attribution functions maintain the optimal policy as the original
reward.
reinforcement learning from human feedback (RLHF) large language model (LLM) alignment scalar rewards final token token-level credit assignment reward shaping optimization problem reward-shaping function SHAP LIME per-token rewards reward model bilevel optimization framework Bayesian Optimization policy training noise from the token reward estimates token-level reward attribution downstream tasks optimal policy feature additive attribution functions