SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn
  Tool-Integrated Reasoning
Large Language Models (LLMs) can significantly improve their reasoning
capabilities by interacting with external tools, a paradigm known as
Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios
using Reinforcement Learning (RL) is often hindered by training instability and
performance collapse. We identify that such instability is primarily caused by
a distributional drift from external tool feedback, leading to the generation
of low-probability tokens. This issue compounds over successive turns, causing
catastrophic gradient norm explosions that derail the training process. To
address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that
stabilizes multi-turn TIR training. Its core strategy is to identify and filter
out trajectories containing void turns, i.e., turns that yield neither a code
block nor a final answer. By removing these problematic trajectories from the
policy update, SimpleTIR effectively blocks the harmful, high-magnitude
gradients, thus stabilizing the learning dynamics. Extensive experiments show
that SimpleTIR achieves state-of-the-art performance on challenging math
reasoning benchmarks, notably elevating the AIME24 score from a text-only
baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model.
Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR
encourages the model to discover diverse and sophisticated reasoning patterns,
such as self-correction and cross-validation.
68b7b9f8295f15ff6091130d
Zhenghai Xue
claimed_verified
2025-09-05T14:16:41.635Z
64c38b3413dc689c2f12f03f
/avatars/d49bf90bb6860189a761b9f5773c09fc.svg
Zhenghai Xue
user
ZhenghaiXue
68b7b9f8295f15ff6091130e
Longtao Zheng
claimed_verified
2025-09-03T19:25:45.985Z
63db5dc49f2687298a1547bf
https://cdn-avatars.huggingface.co/v1/production/uploads/63db5dc49f2687298a1547bf/xVFi0kRkYud191cQgma16.jpeg
Longtao Zheng
user
ltzheng
68b7b9f8295f15ff6091130f
Qian Liu
claimed_verified
2025-09-03T19:25:48.069Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
68b7b9f8295f15ff60911310
Yingru Li
claimed_verified
2025-09-03T08:27:12.625Z
67277d20eebb94a257cd6925
/avatars/c1f714b59fecb53770e28920c0c267cb.svg
Yingru Li
user
R1ch0rd
68b7b9f8295f15ff60911311
Xiaosen Zheng
claimed_verified
2025-09-03T08:27:15.115Z
6274a2315d12b3a734adebc9
https://cdn-avatars.huggingface.co/v1/production/uploads/6274a2315d12b3a734adebc9/zLQDbszAvWh0F2BjdKock.jpeg
Xiaosen Zheng
user
xszheng2020
68b7b9f8295f15ff60911312
Zejun Ma
68b7b9f8295f15ff60911313
Bo An
SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn
  Tool-Integrated Reasoning
Large Language Models (LLMs) can significantly improve their reasoning
capabilities by interacting with external tools, a paradigm known as
Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios
using Reinforcement Learning (RL) is often hindered by training instability and
performance collapse. We identify that such instability is primarily caused by
a distributional drift from external tool feedback, leading to the generation
of low-probability tokens. This issue compounds over successive turns, causing
catastrophic gradient norm explosions that derail the training process. To
address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that
stabilizes multi-turn TIR training. Its core strategy is to identify and filter
out trajectories containing void turns, i.e., turns that yield neither a code
block nor a final answer. By removing these problematic trajectories from the
policy update, SimpleTIR effectively blocks the harmful, high-magnitude
gradients, thus stabilizing the learning dynamics. Extensive experiments show
that SimpleTIR achieves state-of-the-art performance on challenging math
reasoning benchmarks, notably elevating the AIME24 score from a text-only
baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model.
Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR
encourages the model to discover diverse and sophisticated reasoning patterns,
such as self-correction and cross-validation.
SimpleTIR stabilizes multi-turn Tool-Integrated Reasoning training by filtering out void turns, achieving state-of-the-art performance on math reasoning benchmarks.
Tool-Integrated Reasoning TIR Reinforcement Learning RL distributional drift low-probability tokens gradient norm explosions SimpleTIR void turns policy update AIME24 Qwen2.5-7B self-correction cross-validation