VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated
success in enhancing LLM reasoning capabilities, but remains limited to
single-turn interactions without tool integration. While recent Agentic
Reinforcement Learning with Tool use (ARLT) approaches have emerged to address
multi-turn tool interactions, existing works develop task-specific codebases
that suffer from fragmentation, synchronous execution bottlenecks, and limited
extensibility across domains. These inefficiencies hinder broader community
adoption and algorithmic innovation. We introduce VerlTool, a unified and
modular framework that addresses these limitations through systematic design
principles. VerlTool provides four key contributions: (1) upstream alignment
with VeRL ensuring compatibility and simplified maintenance, (2) unified tool
management via standardized APIs supporting diverse modalities including code
execution, search, SQL databases, and vision processing, (3) asynchronous
rollout execution achieving near 2times speedup by eliminating
synchronization bottlenecks, and (4) comprehensive evaluation demonstrating
competitive performance across 6 ARLT domains. Our framework formalizes ARLT as
multi-turn trajectories with multi-modal observation tokens (text/image/video),
extending beyond single-turn RLVR paradigms. We train and evaluate models on
mathematical reasoning, knowledge QA, SQL generation, visual reasoning, web
search, and software engineering tasks, achieving results comparable to
specialized systems while providing unified training infrastructure. The
modular plugin architecture enables rapid tool integration requiring only
lightweight Python definitions, significantly reducing development overhead and
providing a scalable foundation for tool-augmented RL research. Our code is
open-sourced at https://github.com/TIGER-AI-Lab/verl-tool.
68b7d137295f15ff60911475
Dongfu Jiang
claimed_verified
2025-09-03T08:26:13.620Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
68b7d137295f15ff60911476
Yi Lu
claimed_verified
2025-09-03T19:25:21.931Z
65095591cc02352e1e106d0a
/avatars/55bb1dcc31fca1659bcea57d15da50e5.svg
Tom Lu
user
eigentom
68b7d137295f15ff60911477
Zhuofeng Li
claimed_verified
2025-09-03T19:25:16.061Z
66349404f2c753240d02952a
https://cdn-avatars.huggingface.co/v1/production/uploads/66349404f2c753240d02952a/xKBKicwyk7BoOITQPwBJn.png
ZhuofengLi
user
ZhuofengLi
68b7d137295f15ff60911478
Zhiheng Lyu
68b7d137295f15ff60911479
Ping Nie
claimed_verified
2025-09-04T19:36:40.468Z
65358802a920f38780b3248a
/avatars/9415510b598079973c2b0436ad12db9c.svg
Ping Nie
user
pingnieuk
68b7d137295f15ff6091147a
Haozhe Wang
claimed_verified
2025-09-03T08:26:11.647Z
65bf52f0259bc6caeb74f8bf
/avatars/b38392e954466df784a5760ded5df804.svg
Haozhe Wang
user
JasperHaozhe
68b7d137295f15ff6091147b
Alex Su
68b7d137295f15ff6091147c
Hui Chen
claimed_verified
2025-09-03T19:25:18.966Z
61166c4328c98bfd5b92e7c5
/avatars/f4bb0f0cc2c5b84428c28bddaa479b61.svg
Hui Chen
user
chchenhui
68b7d137295f15ff6091147d
Kai Zou
68b7d137295f15ff6091147e
Chao Du
68b7d137295f15ff6091147f
Tianyu Pang
68b7d137295f15ff60911480
Wenhu Chen
VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated
success in enhancing LLM reasoning capabilities, but remains limited to
single-turn interactions without tool integration. While recent Agentic
Reinforcement Learning with Tool use (ARLT) approaches have emerged to address
multi-turn tool interactions, existing works develop task-specific codebases
that suffer from fragmentation, synchronous execution bottlenecks, and limited
extensibility across domains. These inefficiencies hinder broader community
adoption and algorithmic innovation. We introduce VerlTool, a unified and
modular framework that addresses these limitations through systematic design
principles. VerlTool provides four key contributions: (1) upstream alignment
with VeRL ensuring compatibility and simplified maintenance, (2) unified tool
management via standardized APIs supporting diverse modalities including code
execution, search, SQL databases, and vision processing, (3) asynchronous
rollout execution achieving near 2times speedup by eliminating
synchronization bottlenecks, and (4) comprehensive evaluation demonstrating
competitive performance across 6 ARLT domains. Our framework formalizes ARLT as
multi-turn trajectories with multi-modal observation tokens (text/image/video),
extending beyond single-turn RLVR paradigms. We train and evaluate models on
mathematical reasoning, knowledge QA, SQL generation, visual reasoning, web
search, and software engineering tasks, achieving results comparable to
specialized systems while providing unified training infrastructure. The
modular plugin architecture enables rapid tool integration requiring only
lightweight Python definitions, significantly reducing development overhead and
providing a scalable foundation for tool-augmented RL research. Our code is
open-sourced at https://github.com/TIGER-AI-Lab/verl-tool.
VerlTool is a unified and modular framework for Agentic Reinforcement Learning with Tool use, addressing inefficiencies in existing approaches and providing competitive performance across multiple domains.
Reinforcement Learning with Verifiable Rewards Agentic Reinforcement Learning with Tool use VerlTool upstream alignment VeRL standardized APIs asynchronous rollout execution multi-turn trajectories multi-modal observation tokens mathematical reasoning knowledge QA SQL generation visual reasoning web search software engineering modular plugin architecture