SSRL: Self-Search Reinforcement Learning
We investigate the potential of large language models (LLMs) to serve as
efficient simulators for agentic search tasks in reinforcement learning (RL),
thereby reducing dependence on costly interactions with external search
engines. To this end, we first quantify the intrinsic search capability of LLMs
via structured prompting and repeated sampling, which we term Self-Search. Our
results reveal that LLMs exhibit strong scaling behavior with respect to the
inference budget, achieving high pass@k on question-answering benchmarks,
including the challenging BrowseComp task. Building on these observations, we
introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability
through format-based and rule-based rewards. SSRL enables models to iteratively
refine their knowledge utilization internally, without requiring access to
external tools. Empirical evaluations demonstrate that SSRL-trained policy
models provide a cost-effective and stable environment for search-driven RL
training, reducing reliance on external search engines and facilitating robust
sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world
knowledge that can be effectively elicited to achieve high performance; 2) SSRL
demonstrates the potential of leveraging internal knowledge to reduce
hallucination; 3) SSRL-trained models integrate seamlessly with external search
engines without additional effort. Our findings highlight the potential of LLMs
to support more scalable RL agent training.
68a2e369a4caabb4320e64da
Yuchen Fan
claimed_verified
2025-08-25T08:27:52.135Z
672c2d7816766a76a747b7b5
/avatars/12c7b26d2b81721ccac3a5c71e32a1a1.svg
Yuchen Fan
user
yuchenFan
68a2e369a4caabb4320e64db
Kaiyan Zhang
claimed_verified
2025-08-19T08:28:47.455Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
68a2e369a4caabb4320e64dc
Heng Zhou
claimed_verified
2025-08-19T10:48:58.909Z
660d17d6c9be0dcd31a30b3d
/avatars/3743fe9b695c488ebe33f0d8fd607a8a.svg
Zhou Heng
user
henggg
68a2e369a4caabb4320e64dd
Yuxin Zuo
claimed_verified
2025-09-12T16:22:21.408Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
user
yuxinzuo
68a2e369a4caabb4320e64de
Yanxu Chen
68a2e369a4caabb4320e64df
Yu Fu
68a2e369a4caabb4320e64e0
Xinwei Long
68a2e369a4caabb4320e64e1
Xuekai Zhu
claimed_verified
2025-09-19T06:50:58.347Z
647ffddeb82adfa7cc1a10d9
/avatars/26aa168d6b2068298ebb16584aa52b6c.svg
zhu
user
xuekai
68a2e369a4caabb4320e64e2
Che Jiang
68a2e369a4caabb4320e64e3
Yuchen Zhang
claimed_verified
2025-08-21T09:16:22.912Z
66e3f8fb5d97b5bb46923444
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DW806I00-00oQAYvD4ocQ.png
Yuchen Zhang
user
YucZhang2003
68a2e369a4caabb4320e64e4
Li Kang
68a2e369a4caabb4320e64e5
Gang Chen
68a2e369a4caabb4320e64e6
Cheng Huang
68a2e369a4caabb4320e64e7
Zhizhou He
68a2e369a4caabb4320e64e8
Bingning Wang
68a2e369a4caabb4320e64e9
Lei Bai
68a2e369a4caabb4320e64ea
Ning Ding
claimed_verified
2025-09-13T14:57:16.924Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
68a2e369a4caabb4320e64eb
Bowen Zhou
SSRL: Self-Search Reinforcement Learning
We investigate the potential of large language models (LLMs) to serve as
efficient simulators for agentic search tasks in reinforcement learning (RL),
thereby reducing dependence on costly interactions with external search
engines. To this end, we first quantify the intrinsic search capability of LLMs
via structured prompting and repeated sampling, which we term Self-Search. Our
results reveal that LLMs exhibit strong scaling behavior with respect to the
inference budget, achieving high pass@k on question-answering benchmarks,
including the challenging BrowseComp task. Building on these observations, we
introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability
through format-based and rule-based rewards. SSRL enables models to iteratively
refine their knowledge utilization internally, without requiring access to
external tools. Empirical evaluations demonstrate that SSRL-trained policy
models provide a cost-effective and stable environment for search-driven RL
training, reducing reliance on external search engines and facilitating robust
sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world
knowledge that can be effectively elicited to achieve high performance; 2) SSRL
demonstrates the potential of leveraging internal knowledge to reduce
hallucination; 3) SSRL-trained models integrate seamlessly with external search
engines without additional effort. Our findings highlight the potential of LLMs
to support more scalable RL agent training.
LLMs can serve as efficient simulators for RL tasks by leveraging internal knowledge, reducing reliance on external search engines and improving sim-to-real transfer.
large language models LLMs reinforcement learning RL Self-Search pass@k BrowseComp Self-Search RL SSRL format-based rewards rule-based rewards hallucination