START: Self-taught Reasoner with Tools
Large reasoning models (LRMs) like OpenAI-o1 and DeepSeek-R1 have
demonstrated remarkable capabilities in complex reasoning tasks through the
utilization of long Chain-of-thought (CoT). However, these models often suffer
from hallucinations and inefficiencies due to their reliance solely on internal
reasoning processes. In this paper, we introduce START (Self-Taught Reasoner
with Tools), a novel tool-integrated long CoT reasoning LLM that significantly
enhances reasoning capabilities by leveraging external tools. Through code
execution, START is capable of performing complex computations, self-checking,
exploring diverse methods, and self-debugging, thereby addressing the
limitations of LRMs. The core innovation of START lies in its self-learning
framework, which comprises two key techniques: 1) Hint-infer: We demonstrate
that inserting artificially designed hints (e.g., ``Wait, maybe using Python
here is a good idea.'') during the inference process of a LRM effectively
stimulates its ability to utilize external tools without the need for any
demonstration data. Hint-infer can also serve as a simple and effective
sequential test-time scaling method; 2) Hint Rejection Sampling Fine-Tuning
(Hint-RFT): Hint-RFT combines Hint-infer and RFT by scoring, filtering, and
modifying the reasoning trajectories with tool invocation generated by a LRM
via Hint-infer, followed by fine-tuning the LRM. Through this framework, we
have fine-tuned the QwQ-32B model to achieve START. On PhD-level science QA
(GPQA), competition-level math benchmarks (AMC23, AIME24, AIME25), and the
competition-level code benchmark (LiveCodeBench), START achieves accuracy rates
of 63.6%, 95.0%, 66.7%, 47.1%, and 47.3%, respectively. It significantly
outperforms the base QwQ-32B and achieves performance comparable to the
state-of-the-art open-weight model R1-Distill-Qwen-32B and the proprietary
model o1-Preview.
67ca670d3e81e3344dc4c2d9
Chengpeng Li
admin_assigned
2025-03-07T09:12:37.350Z
65294b334d7cf551ac50d6a6
/avatars/75d21e20b711b871616ef3850bb900b7.svg
ChengpengLi
user
ChengpengLi
67ca670d3e81e3344dc4c2da
Mingfeng Xue
admin_assigned
2025-03-07T09:12:28.354Z
5f8946925d083370c711f296
/avatars/14246aae3b1f8b7ad050f8ff2c8b260e.svg
Mingfeng Xue
user
mingfengxue
67ca670d3e81e3344dc4c2db
Zhenru Zhang
admin_assigned
2025-03-07T09:12:48.194Z
64704e973601bb7b06643e98
/avatars/52e51f4d1be6769e4397b8be2799cf32.svg
Zhenru Zhang
user
Zhenru
67ca670d3e81e3344dc4c2dc
Jiaxi Yang
admin_assigned
2025-03-07T09:12:57.082Z
646df403ad20c6fa4f30b7ec
https://cdn-avatars.huggingface.co/v1/production/uploads/646df403ad20c6fa4f30b7ec/Q64-XMghOcBoo3itZDGYA.jpeg
Jiaxi Yang
user
jx-yang
67ca670d3e81e3344dc4c2dd
Beichen Zhang
admin_assigned
2025-03-07T09:13:11.641Z
64b93578ee257c3a4cfceed1
/avatars/e6188562254f75a09b4048b800860016.svg
Beichen Zhang
user
BeichenZhang
67ca670d3e81e3344dc4c2de
Xiang Wang
67ca670d3e81e3344dc4c2df
Bowen Yu
admin_assigned
2025-03-07T09:13:30.530Z
6583ab7983a9e1460c67d876
/avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
bowen
user
bowenYu
67ca670d3e81e3344dc4c2e0
Binyuan Hui
admin_assigned
2025-03-07T09:13:37.341Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
67ca670d3e81e3344dc4c2e1
Junyang Lin
admin_assigned
2025-03-07T09:13:44.084Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
67ca670d3e81e3344dc4c2e2
Dayiheng Liu
admin_assigned
2025-03-07T09:13:52.711Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
START: Self-taught Reasoner with Tools
Large reasoning models (LRMs) like OpenAI-o1 and DeepSeek-R1 have
demonstrated remarkable capabilities in complex reasoning tasks through the
utilization of long Chain-of-thought (CoT). However, these models often suffer
from hallucinations and inefficiencies due to their reliance solely on internal
reasoning processes. In this paper, we introduce START (Self-Taught Reasoner
with Tools), a novel tool-integrated long CoT reasoning LLM that significantly
enhances reasoning capabilities by leveraging external tools. Through code
execution, START is capable of performing complex computations, self-checking,
exploring diverse methods, and self-debugging, thereby addressing the
limitations of LRMs. The core innovation of START lies in its self-learning
framework, which comprises two key techniques: 1) Hint-infer: We demonstrate
that inserting artificially designed hints (e.g., ``Wait, maybe using Python
here is a good idea.'') during the inference process of a LRM effectively
stimulates its ability to utilize external tools without the need for any
demonstration data. Hint-infer can also serve as a simple and effective
sequential test-time scaling method; 2) Hint Rejection Sampling Fine-Tuning
(Hint-RFT): Hint-RFT combines Hint-infer and RFT by scoring, filtering, and
modifying the reasoning trajectories with tool invocation generated by a LRM
via Hint-infer, followed by fine-tuning the LRM. Through this framework, we
have fine-tuned the QwQ-32B model to achieve START. On PhD-level science QA
(GPQA), competition-level math benchmarks (AMC23, AIME24, AIME25), and the
competition-level code benchmark (LiveCodeBench), START achieves accuracy rates
of 63.6%, 95.0%, 66.7%, 47.1%, and 47.3%, respectively. It significantly
outperforms the base QwQ-32B and achieves performance comparable to the
state-of-the-art open-weight model R1-Distill-Qwen-32B and the proprietary
model o1-Preview.
START integrates external tools into large reasoning models to enhance capabilities, using techniques like Hint-infer and Hint Rejection Sampling Fine-Tuning, achieving high performance across various benchmarks.
Large reasoning models Chain-of-thought CoT self-checking self-debugging self-learning framework Hint-infer Hint Rejection Sampling Fine-Tuning Hint-RFT QwQ-32B PhD-level science QA GPQA AMC23 AIME24 AIME25 LiveCodeBench R1-Distill-Qwen-32B o1-Preview