Every Question Has Its Own Value: Reinforcement Learning with Explicit
  Human Values
We propose Reinforcement Learning with Explicit Human Values (RLEV), a method
that aligns Large Language Model (LLM) optimization directly with quantifiable
human value signals. While Reinforcement Learning with Verifiable Rewards
(RLVR) effectively trains models in objective domains using binary correctness
rewards, it overlooks that not all tasks are equally significant. RLEV extends
this framework by incorporating human-defined value signals directly into the
reward function. Using exam-style data with explicit ground-truth value labels,
RLEV consistently outperforms correctness-only baselines across multiple RL
algorithms and model scales. Crucially, RLEV policies not only improve
value-weighted accuracy but also learn a value-sensitive termination policy:
concise for low-value prompts, thorough for high-value ones. We demonstrate
this behavior stems from value-weighted gradient amplification on
end-of-sequence tokens. Ablation studies confirm the gain is causally linked to
value alignment. RLEV remains robust under noisy value signals, such as
difficulty-based labels, demonstrating that optimizing for an explicit utility
function offers a practical path to aligning LLMs with human priorities.
68fad821f158a71c5a2f57f4
Dian Yu
68fad821f158a71c5a2f57f5
Yulai Zhao
claimed_verified
2025-10-24T16:12:37.233Z
6575171654d1749612e21eed
/avatars/c032c1b942b3cb9450a49db88fce5c70.svg
Yulai Zhao
user
sarosavo
68fad821f158a71c5a2f57f6
Kishan Panaganti
claimed_verified
2025-10-24T16:12:34.051Z
68266b5261ed4d89177c3612
/avatars/e41ef277047334174eca408aba2a63db.svg
Kishan
user
kishanpb
68fad821f158a71c5a2f57f7
Linfeng Song
68fad821f158a71c5a2f57f8
Haitao Mi
68fad821f158a71c5a2f57f9
Dong Yu
Every Question Has Its Own Value: Reinforcement Learning with Explicit
  Human Values
We propose Reinforcement Learning with Explicit Human Values (RLEV), a method
that aligns Large Language Model (LLM) optimization directly with quantifiable
human value signals. While Reinforcement Learning with Verifiable Rewards
(RLVR) effectively trains models in objective domains using binary correctness
rewards, it overlooks that not all tasks are equally significant. RLEV extends
this framework by incorporating human-defined value signals directly into the
reward function. Using exam-style data with explicit ground-truth value labels,
RLEV consistently outperforms correctness-only baselines across multiple RL
algorithms and model scales. Crucially, RLEV policies not only improve
value-weighted accuracy but also learn a value-sensitive termination policy:
concise for low-value prompts, thorough for high-value ones. We demonstrate
this behavior stems from value-weighted gradient amplification on
end-of-sequence tokens. Ablation studies confirm the gain is causally linked to
value alignment. RLEV remains robust under noisy value signals, such as
difficulty-based labels, demonstrating that optimizing for an explicit utility
function offers a practical path to aligning LLMs with human priorities.
RLEV, a reinforcement learning method, aligns LLM optimization with human value signals, improving value-weighted accuracy and learning value-sensitive termination policies.
Reinforcement Learning with Explicit Human Values (RLEV) Large Language Model (LLM) Reinforcement Learning with Verifiable Rewards (RLVR) value signals reward function exam-style data value-weighted accuracy value-sensitive termination policy gradient amplification end-of-sequence tokens ablation studies value alignment noisy value signals utility function