Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
Reinforcement Learning (RL) has proven highly effective at enhancing the
complex reasoning abilities of Large Language Models (LLMs), yet underlying
mechanisms driving this success remain largely opaque. Our analysis reveals
that puzzling phenomena like ``aha moments", ``length-scaling'' and entropy
dynamics are not disparate occurrences but hallmarks of an emergent reasoning
hierarchy, akin to the separation of high-level strategic planning from
low-level procedural execution in human cognition. We uncover a compelling
two-phase dynamic: initially, a model is constrained by procedural correctness
and must improve its low-level skills. The learning bottleneck then decisively
shifts, with performance gains being driven by the exploration and mastery of
high-level strategic planning. This insight exposes a core inefficiency in
prevailing RL algorithms like GRPO, which apply optimization pressure
agnostically and dilute the learning signal across all tokens. To address this,
we propose HIerarchy-Aware Credit Assignment (HICRA), an algorithm that
concentrates optimization efforts on high-impact planning tokens. HICRA
significantly outperforms strong baselines, demonstrating that focusing on this
strategic bottleneck is key to unlocking advanced reasoning. Furthermore, we
validate semantic entropy as a superior compass for measuring strategic
exploration over misleading metrics such as token-level entropy.
68baebf4e41474e84d6ddd07
Haozhe Wang
68baebf4e41474e84d6ddd08
Qixin Xu
claimed_verified
2025-09-09T13:49:47.551Z
6680f0b20b72be136708af26
/avatars/5d8fd5be0cf94e246b46abb9d3cc8f5c.svg
XuQixin
user
Racktic
68baebf4e41474e84d6ddd09
Che Liu
68baebf4e41474e84d6ddd0a
Junhong Wu
68baebf4e41474e84d6ddd0b
Fangzhen Lin
68baebf4e41474e84d6ddd0c
Wenhu Chen
Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
Reinforcement Learning (RL) has proven highly effective at enhancing the
complex reasoning abilities of Large Language Models (LLMs), yet underlying
mechanisms driving this success remain largely opaque. Our analysis reveals
that puzzling phenomena like ``aha moments", ``length-scaling'' and entropy
dynamics are not disparate occurrences but hallmarks of an emergent reasoning
hierarchy, akin to the separation of high-level strategic planning from
low-level procedural execution in human cognition. We uncover a compelling
two-phase dynamic: initially, a model is constrained by procedural correctness
and must improve its low-level skills. The learning bottleneck then decisively
shifts, with performance gains being driven by the exploration and mastery of
high-level strategic planning. This insight exposes a core inefficiency in
prevailing RL algorithms like GRPO, which apply optimization pressure
agnostically and dilute the learning signal across all tokens. To address this,
we propose HIerarchy-Aware Credit Assignment (HICRA), an algorithm that
concentrates optimization efforts on high-impact planning tokens. HICRA
significantly outperforms strong baselines, demonstrating that focusing on this
strategic bottleneck is key to unlocking advanced reasoning. Furthermore, we
validate semantic entropy as a superior compass for measuring strategic
exploration over misleading metrics such as token-level entropy.
Reinforcement Learning enhances LLM reasoning through a two-phase process involving procedural correctness and strategic planning, with HICRA algorithm focusing on high-impact planning tokens to improve performance.
Reinforcement Learning Large Language Models aha moments length-scaling entropy dynamics reasoning hierarchy procedural correctness high-level strategic planning low-level procedural execution GRPO HIerarchy-Aware Credit Assignment HICRA semantic entropy token-level entropy