Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale
  Asynchronous RL
Recent advancements in LLM-based agents have demonstrated remarkable
capabilities in handling complex, knowledge-intensive tasks by integrating
external tools. Among diverse choices of tools, search tools play a pivotal
role in accessing vast external knowledge. However, open-source agents still
fall short of achieving expert-level Search Intelligence, the ability to
resolve ambiguous queries, generate precise searches, analyze results, and
conduct thorough exploration. Existing approaches fall short in scalability,
efficiency, and data quality. For example, small turn limits in existing online
RL methods, e.g. <=10, restrict complex strategy learning. This paper
introduces ASearcher, an open-source project for large-scale RL training of
search agents. Our key contributions include: (1) Scalable fully asynchronous
RL training that enables long-horizon search while maintaining high training
efficiency. (2) A prompt-based LLM agent that autonomously synthesizes
high-quality and challenging QAs, creating a large-scale QA dataset. Through RL
training, our prompt-based QwQ-32B agent achieves substantial improvements,
with 46.7% and 20.8% Avg@4 gains on xBench and GAIA, respectively. Notably, our
agent exhibits extreme long-horizon search, with tool calls exceeding 40 turns
and output tokens exceeding 150k during training time. With a simple agent
design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 42.1 on
xBench and 52.8 on GAIA, surpassing existing open-source 32B agents. We
open-source our models, training data, and codes in
https://github.com/inclusionAI/ASearcher.
689bf8c2fab6fdd2e52ac825
Jiaxuan Gao
689bf8c2fab6fdd2e52ac826
Wei Fu
689bf8c2fab6fdd2e52ac827
Minyang Xie
689bf8c2fab6fdd2e52ac828
Shusheng Xu
689bf8c2fab6fdd2e52ac829
Chuyi He
689bf8c2fab6fdd2e52ac82a
Zhiyu Mei
689bf8c2fab6fdd2e52ac82b
Banghua Zhu
689bf8c2fab6fdd2e52ac82c
Yi Wu
Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale
  Asynchronous RL
Recent advancements in LLM-based agents have demonstrated remarkable
capabilities in handling complex, knowledge-intensive tasks by integrating
external tools. Among diverse choices of tools, search tools play a pivotal
role in accessing vast external knowledge. However, open-source agents still
fall short of achieving expert-level Search Intelligence, the ability to
resolve ambiguous queries, generate precise searches, analyze results, and
conduct thorough exploration. Existing approaches fall short in scalability,
efficiency, and data quality. For example, small turn limits in existing online
RL methods, e.g. <=10, restrict complex strategy learning. This paper
introduces ASearcher, an open-source project for large-scale RL training of
search agents. Our key contributions include: (1) Scalable fully asynchronous
RL training that enables long-horizon search while maintaining high training
efficiency. (2) A prompt-based LLM agent that autonomously synthesizes
high-quality and challenging QAs, creating a large-scale QA dataset. Through RL
training, our prompt-based QwQ-32B agent achieves substantial improvements,
with 46.7% and 20.8% Avg@4 gains on xBench and GAIA, respectively. Notably, our
agent exhibits extreme long-horizon search, with tool calls exceeding 40 turns
and output tokens exceeding 150k during training time. With a simple agent
design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 42.1 on
xBench and 52.8 on GAIA, surpassing existing open-source 32B agents. We
open-source our models, training data, and codes in
https://github.com/inclusionAI/ASearcher.
ASearcher is an open-source project that uses scalable asynchronous RL training to enhance search agents, achieving high performance on QA tasks with long-horizon search capabilities.
LLM-based agents search tools Search Intelligence open-source agents online RL methods fully asynchronous RL training prompt-based LLM agent QA dataset xBench GAIA Avg@4 tool calls output tokens ASearcher-Web-QwQ