From Uniform to Heterogeneous: Tailoring Policy Optimization to Every
  Token's Nature
Reinforcement Learning has emerged as the fundamental technique for enhancing
reasoning in LLMs. However, existing algorithms apply uniform optimization to
all tokens, ignoring their different roles in reasoning process. To address
this limitation, we introduce Heterogeneous Adaptive Policy Optimization
(HAPO), a comprehensive token-aware algorithm that dynamically adapts
optimization based on token entropy. For rollout sampling, we propose Adaptive
Temperature Sampling, which adjusts sampling temperature in real time,
promoting exploration at high-entropy tokens while preserving coherence at
low-entropy ones. For advantage calculation, we introduce Token Level Group
Average that normalizes advantages at token level, jointly accounting for
sequence-length as in token-mean loss while preserving non-biased treatment. We
then develop Differential Advantage Redistribution that leverages entropy and
importance ratios to modulate rewards-adjusting updates for tokens with clear
signals. For clipping loss, we design Asymmetric Adaptive Clipping, allowing
aggressive probability reduction for noisy low-entropy tokens while enabling
exploration for high-entropy tokens. Through systematic investigation between
entropy and training dynamics, we embedded token-level treatment into every
stages to achieve fine-grained control. Extensive experiments demonstrate that
HAPO consistently outperforms DAPO across multiple model scales. Our code can
be found in https://github.com/starriver030515/HAPO.
68d217261ca7156988a8edb3
Zheng Liu
claimed_verified
2025-09-23T10:05:44.086Z
6625ef13605f46d05c1d0031
/avatars/22f201dca35e43013cb593884516e96c.svg
Zheng Liu
user
starriver030515
68d217261ca7156988a8edb4
Mengjie Liu
admin_assigned
2025-09-23T14:34:50.983Z
672dbf1bec84b9c33412488f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/V6o6K28ydpkaS9XFA4Ac3.png
Mengjie Liu
user
Balalauuoo
68d217261ca7156988a8edb5
Siwei Wen
68d217261ca7156988a8edb6
Mengzhang Cai
68d217261ca7156988a8edb7
Bin Cui
68d217261ca7156988a8edb8
Conghui He
admin_assigned
2025-09-23T14:35:08.157Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
68d217261ca7156988a8edb9
Wentao Zhang
From Uniform to Heterogeneous: Tailoring Policy Optimization to Every
  Token's Nature
Reinforcement Learning has emerged as the fundamental technique for enhancing
reasoning in LLMs. However, existing algorithms apply uniform optimization to
all tokens, ignoring their different roles in reasoning process. To address
this limitation, we introduce Heterogeneous Adaptive Policy Optimization
(HAPO), a comprehensive token-aware algorithm that dynamically adapts
optimization based on token entropy. For rollout sampling, we propose Adaptive
Temperature Sampling, which adjusts sampling temperature in real time,
promoting exploration at high-entropy tokens while preserving coherence at
low-entropy ones. For advantage calculation, we introduce Token Level Group
Average that normalizes advantages at token level, jointly accounting for
sequence-length as in token-mean loss while preserving non-biased treatment. We
then develop Differential Advantage Redistribution that leverages entropy and
importance ratios to modulate rewards-adjusting updates for tokens with clear
signals. For clipping loss, we design Asymmetric Adaptive Clipping, allowing
aggressive probability reduction for noisy low-entropy tokens while enabling
exploration for high-entropy tokens. Through systematic investigation between
entropy and training dynamics, we embedded token-level treatment into every
stages to achieve fine-grained control. Extensive experiments demonstrate that
HAPO consistently outperforms DAPO across multiple model scales. Our code can
be found in https://github.com/starriver030515/HAPO.
Heterogeneous Adaptive Policy Optimization (HAPO) enhances reinforcement learning in LLMs by dynamically adapting token optimization based on entropy, improving performance across various model scales.
Heterogeneous Adaptive Policy Optimization HAPO token-aware algorithm token entropy Adaptive Temperature Sampling Token Level Group Average Differential Advantage Redistribution Asymmetric Adaptive Clipping