Demystifying Reinforcement Learning in Agentic Reasoning
Recently, the emergence of agentic RL has showcased that RL could also
effectively improve the agentic reasoning ability of LLMs, yet the key design
principles and optimal practices remain unclear. In this work, we conduct a
comprehensive and systematic investigation to demystify reinforcement learning
in agentic reasoning from three key perspectives: data, algorithm, and
reasoning mode. We highlight our key insights: (i) Replacing stitched synthetic
trajectories with real end-to-end tool-use trajectories yields a far stronger
SFT initialization; high-diversity, model-aware datasets sustain exploration
and markedly improve RL performance. (ii) Exploration-friendly techniques are
crucial for agentic RL, such as clip higher, overlong reward shaping, and
maintaining adequate policy entropy could improve the training efficiency.
(iii) A deliberative strategy with fewer tool calls outperforms frequent tool
calls or verbose self-reasoning, improving tool efficiency and final accuracy.
Together, these simple practices consistently enhance agentic reasoning and
training efficiency, achieving strong results on challenging benchmarks with
smaller models, and establishing a practical baseline for future agentic RL
research. Beyond these empirical insights, we further contribute a
high-quality, real end-to-end agentic SFT dataset along with a high-quality RL
dataset, and demonstrate the effectiveness of our insights in boosting the
agentic reasoning ability of LLMs across four challenging benchmarks, including
AIME2024/AIME2025, GPQA-Diamond, and LiveCodeBench-v6. With our recipes,
4B-sized models could also achieve superior agentic reasoning performance
compared to 32B-sized models. Code and models:
https://github.com/Gen-Verse/Open-AgentRL
68edb4e8de1fee572713a782
Zhaochen Yu
68edb4e8de1fee572713a783
Ling Yang
68edb4e8de1fee572713a784
Jiaru Zou
68edb4e8de1fee572713a785
Shuicheng Yan
68edb4e8de1fee572713a786
Mengdi Wang
Demystifying Reinforcement Learning in Agentic Reasoning
Recently, the emergence of agentic RL has showcased that RL could also
effectively improve the agentic reasoning ability of LLMs, yet the key design
principles and optimal practices remain unclear. In this work, we conduct a
comprehensive and systematic investigation to demystify reinforcement learning
in agentic reasoning from three key perspectives: data, algorithm, and
reasoning mode. We highlight our key insights: (i) Replacing stitched synthetic
trajectories with real end-to-end tool-use trajectories yields a far stronger
SFT initialization; high-diversity, model-aware datasets sustain exploration
and markedly improve RL performance. (ii) Exploration-friendly techniques are
crucial for agentic RL, such as clip higher, overlong reward shaping, and
maintaining adequate policy entropy could improve the training efficiency.
(iii) A deliberative strategy with fewer tool calls outperforms frequent tool
calls or verbose self-reasoning, improving tool efficiency and final accuracy.
Together, these simple practices consistently enhance agentic reasoning and
training efficiency, achieving strong results on challenging benchmarks with
smaller models, and establishing a practical baseline for future agentic RL
research. Beyond these empirical insights, we further contribute a
high-quality, real end-to-end agentic SFT dataset along with a high-quality RL
dataset, and demonstrate the effectiveness of our insights in boosting the
agentic reasoning ability of LLMs across four challenging benchmarks, including
AIME2024/AIME2025, GPQA-Diamond, and LiveCodeBench-v6. With our recipes,
4B-sized models could also achieve superior agentic reasoning performance
compared to 32B-sized models. Code and models:
https://github.com/Gen-Verse/Open-AgentRL
Agentic reinforcement learning enhances LLMs' reasoning ability through real datasets, exploration techniques, and a deliberative strategy, achieving strong performance with smaller models.
agentic RL SFT initialization real end-to-end tool-use trajectories high-diversity datasets model-aware datasets exploration-friendly techniques clip higher overlong reward shaping policy entropy deliberative strategy agentic SFT dataset RL dataset AIME2024/AIME2025 GPQA-Diamond LiveCodeBench-v6