Search Self-play: Pushing the Frontier of Agent Capability without
  Supervision
Reinforcement learning with verifiable rewards (RLVR) has become the
mainstream technique for training LLM agents. However, RLVR highly depends on
well-crafted task queries and corresponding ground-truth answers to provide
accurate rewards, which requires massive human efforts and hinders the RL
scaling processes, especially under agentic scenarios. Although a few recent
works explore task synthesis methods, the difficulty of generated agentic tasks
can hardly be controlled to provide effective RL training advantages. To
achieve agentic RLVR with higher scalability, we explore self-play training for
deep search agents, in which the learning LLM utilizes multi-turn search engine
calling and acts simultaneously as both a task proposer and a problem solver.
The task proposer aims to generate deep search queries with well-defined
ground-truth answers and increasing task difficulty. The problem solver tries
to handle the generated search queries and output the correct answer
predictions. To ensure that each generated search query has accurate ground
truth, we collect all the searching results from the proposer's trajectory as
external knowledge, then conduct retrieval-augmentation generation (RAG) to
test whether the proposed query can be correctly answered with all necessary
search documents provided. In this search self-play (SSP) game, the proposer
and the solver co-evolve their agent capabilities through both competition and
cooperation. With substantial experimental results, we find that SSP can
significantly improve search agents' performance uniformly on various
benchmarks without any supervision under both from-scratch and continuous RL
training setups. The code is at https://github.com/Alibaba-Quark/SSP.
68f85c9b7669bcaeecce0d9a
Hongliang Lu
68f85c9b7669bcaeecce0d9b
Yuhang Wen
claimed_verified
2025-10-22T15:43:23.308Z
669ffabefbfc379ab3d8fbb0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xCjsWtFvbZPLRrOv_1zZD.jpeg
Yuhang Wen
user
Necolizer
68f85c9b7669bcaeecce0d9c
Pengyu Cheng
68f85c9b7669bcaeecce0d9d
Ruijin Ding
68f85c9b7669bcaeecce0d9e
Haotian Xu
68f85c9b7669bcaeecce0d9f
Jiaqi Guo
68f85c9b7669bcaeecce0da0
Chutian Wang
68f85c9b7669bcaeecce0da1
Haonan Chen
68f85c9b7669bcaeecce0da2
Xiaoxi Jiang
68f85c9b7669bcaeecce0da3
Guanjun Jiang
Search Self-play: Pushing the Frontier of Agent Capability without
  Supervision
Reinforcement learning with verifiable rewards (RLVR) has become the
mainstream technique for training LLM agents. However, RLVR highly depends on
well-crafted task queries and corresponding ground-truth answers to provide
accurate rewards, which requires massive human efforts and hinders the RL
scaling processes, especially under agentic scenarios. Although a few recent
works explore task synthesis methods, the difficulty of generated agentic tasks
can hardly be controlled to provide effective RL training advantages. To
achieve agentic RLVR with higher scalability, we explore self-play training for
deep search agents, in which the learning LLM utilizes multi-turn search engine
calling and acts simultaneously as both a task proposer and a problem solver.
The task proposer aims to generate deep search queries with well-defined
ground-truth answers and increasing task difficulty. The problem solver tries
to handle the generated search queries and output the correct answer
predictions. To ensure that each generated search query has accurate ground
truth, we collect all the searching results from the proposer's trajectory as
external knowledge, then conduct retrieval-augmentation generation (RAG) to
test whether the proposed query can be correctly answered with all necessary
search documents provided. In this search self-play (SSP) game, the proposer
and the solver co-evolve their agent capabilities through both competition and
cooperation. With substantial experimental results, we find that SSP can
significantly improve search agents' performance uniformly on various
benchmarks without any supervision under both from-scratch and continuous RL
training setups. The code is at https://github.com/Alibaba-Quark/SSP.
Self-play training for deep search agents improves performance through co-evolution of task generation and problem solving without supervision.
reinforcement learning with verifiable rewards RLVR self-play training deep search agents multi-turn search engine calling task proposer problem solver retrieval-augmentation generation RAG search self-play SSP from-scratch continuous RL training