THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical
  Reasoning
Large Language Models (LLMs) have made remarkable progress in mathematical
reasoning, but still continue to struggle with high-precision tasks like
numerical computation and formal symbolic manipulation. Integrating external
tools has emerged as a promising approach to bridge this gap. Despite recent
advances, existing methods struggle with three key challenges: constructing
tool-integrated reasoning data, performing fine-grained optimization, and
enhancing inference. To overcome these limitations, we propose THOR
(Tool-Integrated Hierarchical Optimization via RL). First, we introduce TIRGen,
a multi-agent actor-critic-based pipeline for constructing high-quality
datasets of tool-integrated reasoning paths, aligning with the policy and
generalizing well across diverse models. Second, to perform fine-grained
hierarchical optimization, we introduce an RL strategy that jointly optimizes
for both trajectory-level problem solving and step-level code generation. This
is motivated by our key insight that the success of an intermediate tool call
is a strong predictor of the final answer's correctness. Finally, THOR
incorporates a self-correction mechanism that leverages immediate tool feedback
to dynamically revise erroneous reasoning paths during inference. Our approach
demonstrates strong generalization across diverse models, performing
effectively in both reasoning and non-reasoning models. It further achieves
state-of-the-art performance for models of a similar scale on multiple
mathematical benchmarks, while also delivering consistent improvements on code
benchmarks. Our code will be publicly available at
https://github.com/JingMog/THOR.
68cb674b5a7803ff3be42d0e
Qikai Chang
claimed_verified
2025-09-18T13:27:12.793Z
6310ad26f5acafacccf91ac6
/avatars/b9de109e013f8f181e61c03b6fce47a3.svg
Chang
user
JsingMog
68cb674b5a7803ff3be42d0f
Zhenrong Zhang
68cb674b5a7803ff3be42d10
Pengfei Hu
68cb674b5a7803ff3be42d11
Jiefeng Ma
68cb674b5a7803ff3be42d12
Yicheng Pan
68cb674b5a7803ff3be42d13
Jianshu Zhang
68cb674b5a7803ff3be42d14
Jun Du
68cb674b5a7803ff3be42d15
Quan Liu
68cb674b5a7803ff3be42d16
Jianqing Gao
THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical
  Reasoning
Large Language Models (LLMs) have made remarkable progress in mathematical
reasoning, but still continue to struggle with high-precision tasks like
numerical computation and formal symbolic manipulation. Integrating external
tools has emerged as a promising approach to bridge this gap. Despite recent
advances, existing methods struggle with three key challenges: constructing
tool-integrated reasoning data, performing fine-grained optimization, and
enhancing inference. To overcome these limitations, we propose THOR
(Tool-Integrated Hierarchical Optimization via RL). First, we introduce TIRGen,
a multi-agent actor-critic-based pipeline for constructing high-quality
datasets of tool-integrated reasoning paths, aligning with the policy and
generalizing well across diverse models. Second, to perform fine-grained
hierarchical optimization, we introduce an RL strategy that jointly optimizes
for both trajectory-level problem solving and step-level code generation. This
is motivated by our key insight that the success of an intermediate tool call
is a strong predictor of the final answer's correctness. Finally, THOR
incorporates a self-correction mechanism that leverages immediate tool feedback
to dynamically revise erroneous reasoning paths during inference. Our approach
demonstrates strong generalization across diverse models, performing
effectively in both reasoning and non-reasoning models. It further achieves
state-of-the-art performance for models of a similar scale on multiple
mathematical benchmarks, while also delivering consistent improvements on code
benchmarks. Our code will be publicly available at
https://github.com/JingMog/THOR.
THOR, a tool-integrated hierarchical optimization framework using RL, enhances mathematical reasoning and code generation by constructing high-quality datasets, optimizing reasoning paths, and correcting errors during inference.
Large Language Models mathematical reasoning numerical computation formal symbolic manipulation tool-integrated reasoning multi-agent actor-critic TIRGen hierarchical optimization RL strategy trajectory-level problem solving step-level code generation self-correction mechanism immediate tool feedback reasoning paths generalization mathematical benchmarks code benchmarks