Attention as a Compass: Efficient Exploration for Process-Supervised RL
  in Reasoning Models
Reinforcement Learning (RL) has shown remarkable success in enhancing the
reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL
(PSRL) has emerged as a more effective paradigm compared to outcome-based RL.
However, existing PSRL approaches suffer from limited exploration efficiency,
both in terms of branching positions and sampling. In this paper, we introduce
a novel PSRL framework (AttnRL), which enables efficient exploration for
reasoning models. Motivated by preliminary observations that steps exhibiting
high attention scores correlate with reasoning behaviors, we propose to branch
from positions with high values. Furthermore, we develop an adaptive sampling
strategy that accounts for problem difficulty and historical batch size,
ensuring that the whole training batch maintains non-zero advantage values. To
further improve sampling efficiency, we design a one-step off-policy training
pipeline for PSRL. Extensive experiments on multiple challenging mathematical
reasoning benchmarks demonstrate that our method consistently outperforms prior
approaches in terms of performance and sampling and training efficiency.
68dc8bf34159d1f2418f9a6e
Runze Liu
claimed_verified
2025-10-01T10:22:06.937Z
667187ba9ab144eb3ac43a1b
/avatars/db5558aa1c5160b9aee8b58573271959.svg
Runze Liu
user
RyanLiu112
68dc8bf34159d1f2418f9a6f
Jiakang Wang
68dc8bf34159d1f2418f9a70
Yuling Shi
claimed_verified
2025-10-01T10:22:04.689Z
645b0c3ec35da9c7afd95421
https://cdn-avatars.huggingface.co/v1/production/uploads/645b0c3ec35da9c7afd95421/vYBrCDagHsXAo6J2p-uG0.jpeg
Yuling
user
YerbaPage
68dc8bf34159d1f2418f9a71
Zhihui Xie
68dc8bf34159d1f2418f9a72
Chenxin An
68dc8bf34159d1f2418f9a73
Kaiyan Zhang
68dc8bf34159d1f2418f9a74
Jian Zhao
68dc8bf34159d1f2418f9a75
Xiaodong Gu
68dc8bf34159d1f2418f9a76
Lei Lin
68dc8bf34159d1f2418f9a77
Wenping Hu
68dc8bf34159d1f2418f9a78
Xiu Li
68dc8bf34159d1f2418f9a79
Fuzheng Zhang
68dc8bf34159d1f2418f9a7a
Guorui Zhou
68dc8bf34159d1f2418f9a7b
Kun Gai
Attention as a Compass: Efficient Exploration for Process-Supervised RL
  in Reasoning Models
Reinforcement Learning (RL) has shown remarkable success in enhancing the
reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL
(PSRL) has emerged as a more effective paradigm compared to outcome-based RL.
However, existing PSRL approaches suffer from limited exploration efficiency,
both in terms of branching positions and sampling. In this paper, we introduce
a novel PSRL framework (AttnRL), which enables efficient exploration for
reasoning models. Motivated by preliminary observations that steps exhibiting
high attention scores correlate with reasoning behaviors, we propose to branch
from positions with high values. Furthermore, we develop an adaptive sampling
strategy that accounts for problem difficulty and historical batch size,
ensuring that the whole training batch maintains non-zero advantage values. To
further improve sampling efficiency, we design a one-step off-policy training
pipeline for PSRL. Extensive experiments on multiple challenging mathematical
reasoning benchmarks demonstrate that our method consistently outperforms prior
approaches in terms of performance and sampling and training efficiency.
A novel PSRL framework (AttnRL) enhances exploration efficiency in reasoning models by branching from high attention positions and using an adaptive sampling strategy, outperforming prior methods in mathematical reasoning benchmarks.
Reinforcement Learning Large Language Models Process-Supervised RL PSRL AttnRL attention scores reasoning behaviors adaptive sampling strategy one-step off-policy training mathematical reasoning benchmarks