Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm
  Enables Fine-Grained Policy Optimization
The reasoning pattern of Large language models (LLMs) remains opaque, and
Reinforcement learning (RL) typically applies uniform credit across an entire
generation, blurring the distinction between pivotal and routine steps. This
work positions attention as a privileged substrate that renders the internal
logic of LLMs legible, not merely as a byproduct of computation, but as a
mechanistic blueprint of reasoning itself. We first distinguish attention heads
between locally and globally focused information processing and reveal that
locally focused heads produce a sawtooth pattern near the diagonal indicating
phrasal chunks, while globally focused heads expose tokens that exert broad
downstream influence over future tokens. We formalize these with two metrics:
1) Windowed Average Attention Distance, which measures the extent of backward
attention within a clipped window; 2) Future Attention Influence, which
quantifies a token's global importance as the average attention it receives
from subsequent tokens. Taken together, these signals reveal a recurring
preplan-and-anchor mechanism, where the model first performs a long-range
contextual reference to generate an introductory token, which is immediately
followed by or coincides with a semantic anchor token that organizes subsequent
reasoning. Leveraging these insights, we introduce three novel RL strategies
that dynamically perform targeted credit assignment to critical nodes (preplan
tokens, anchor tokens, and their temporal coupling) and show consistent
performance gains across various reasoning tasks. By aligning optimization with
the model's intrinsic reasoning rhythm, we aim to transform opaque optimization
into an actionable structure-aware process, hoping to offer a potential step
toward more transparent and effective optimization of LLM reasoning.
68f04b7636f8b025381e18bf
Yang Li
claimed_verified
2025-10-16T10:38:35.110Z
6570587b44d0620d221c722b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T-B6q6sqhWAyXM7Hk_qWp.jpeg
Yang Li (SJTU & SII)
user
yangcole
68f04b7636f8b025381e18c0
Zhichen Dong
68f04b7636f8b025381e18c1
Yuhan Sun
68f04b7636f8b025381e18c2
Weixun Wang
68f04b7636f8b025381e18c3
Shaopan Xiong
68f04b7636f8b025381e18c4
Yijia Luo
claimed_verified
2025-10-16T10:38:37.247Z
6731715cacb365a5ccd14258
/avatars/919bd7f51b0b7ac52e55846f8d78c293.svg
YIJIALUO
user
YIJIALUO
68f04b7636f8b025381e18c5
Jiashun Liu
68f04b7636f8b025381e18c6
Han Lu
claimed_verified
2025-10-16T06:39:32.455Z
66825d697b0920f40d16c74d
/avatars/0048c8865a8de94218cde580fe0f8bc9.svg
Han Lu
user
SJTULH
68f04b7636f8b025381e18c7
Jiamang Wang
68f04b7636f8b025381e18c8
Wenbo Su
68f04b7636f8b025381e18c9
Bo Zheng
68f04b7636f8b025381e18ca
Junchi Yan
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm
  Enables Fine-Grained Policy Optimization
The reasoning pattern of Large language models (LLMs) remains opaque, and
Reinforcement learning (RL) typically applies uniform credit across an entire
generation, blurring the distinction between pivotal and routine steps. This
work positions attention as a privileged substrate that renders the internal
logic of LLMs legible, not merely as a byproduct of computation, but as a
mechanistic blueprint of reasoning itself. We first distinguish attention heads
between locally and globally focused information processing and reveal that
locally focused heads produce a sawtooth pattern near the diagonal indicating
phrasal chunks, while globally focused heads expose tokens that exert broad
downstream influence over future tokens. We formalize these with two metrics:
1) Windowed Average Attention Distance, which measures the extent of backward
attention within a clipped window; 2) Future Attention Influence, which
quantifies a token's global importance as the average attention it receives
from subsequent tokens. Taken together, these signals reveal a recurring
preplan-and-anchor mechanism, where the model first performs a long-range
contextual reference to generate an introductory token, which is immediately
followed by or coincides with a semantic anchor token that organizes subsequent
reasoning. Leveraging these insights, we introduce three novel RL strategies
that dynamically perform targeted credit assignment to critical nodes (preplan
tokens, anchor tokens, and their temporal coupling) and show consistent
performance gains across various reasoning tasks. By aligning optimization with
the model's intrinsic reasoning rhythm, we aim to transform opaque optimization
into an actionable structure-aware process, hoping to offer a potential step
toward more transparent and effective optimization of LLM reasoning.
Attention mechanisms in LLMs are analyzed to reveal reasoning patterns, leading to novel RL strategies that improve performance by focusing on critical tokens.
Large language models Reinforcement learning attention heads locally focused globally focused Windowed Average Attention Distance Future Attention Influence preplan tokens anchor tokens targeted credit assignment