Search-R1: Training LLMs to Reason and Leverage Search Engines with
  Reinforcement Learning
Efficiently acquiring external knowledge and up-to-date information is
essential for effective reasoning and text generation in large language models
(LLMs). Retrieval augmentation and tool-use training approaches where a search
engine is treated as a tool lack complex multi-turn retrieval flexibility or
require large-scale supervised data. Prompting advanced LLMs with reasoning
capabilities during inference to use search engines is not optimal, since the
LLM does not learn how to optimally interact with the search engine. This paper
introduces Search-R1, an extension of the DeepSeek-R1 model where the LLM
learns -- solely through reinforcement learning (RL) -- to autonomously
generate (multiple) search queries during step-by-step reasoning with real-time
retrieval. Search-R1 optimizes LLM rollouts with multi-turn search
interactions, leveraging retrieved token masking for stable RL training and a
simple outcome-based reward function. Experiments on seven question-answering
datasets show that Search-R1 improves performance by 26% (Qwen2.5-7B), 21%
(Qwen2.5-3B), and 10% (LLaMA3.2-3B) over SOTA baselines. This paper further
provides empirical insights into RL optimization methods, LLM choices, and
response length dynamics in retrieval-augmented reasoning. The code and model
checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.
67d238ab7d0fc37e671feaa8
Bowen Jin
claimed_verified
2025-03-13T08:24:22.835Z
64b8222749bde5d948104627
/avatars/e179dc68aebae503dcd7ea6b65b4a4b7.svg
Bowen
user
PeterJinGo
67d238ab7d0fc37e671feaa9
Hansi Zeng
admin_assigned
2025-03-14T11:24:04.316Z
64395fa0d45e8db3e28667a4
/avatars/cd050f59e2ac79f02ee01dcbb3be00e5.svg
Hansi Zeng
user
hzeng
67d238ab7d0fc37e671feaaa
Zhenrui Yue
admin_assigned
2025-03-14T11:24:10.545Z
644c4eff9a7c19cfaf0c8f4a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2gHtfhPFjSsnwxVtBnxe2.jpeg
Zhenrui Yue
user
yueeeeeeee2837
67d238ab7d0fc37e671feaab
Dong Wang
67d238ab7d0fc37e671feaac
Hamed Zamani
67d238ab7d0fc37e671feaad
Jiawei Han
Search-R1: Training LLMs to Reason and Leverage Search Engines with
  Reinforcement Learning
Efficiently acquiring external knowledge and up-to-date information is
essential for effective reasoning and text generation in large language models
(LLMs). Retrieval augmentation and tool-use training approaches where a search
engine is treated as a tool lack complex multi-turn retrieval flexibility or
require large-scale supervised data. Prompting advanced LLMs with reasoning
capabilities during inference to use search engines is not optimal, since the
LLM does not learn how to optimally interact with the search engine. This paper
introduces Search-R1, an extension of the DeepSeek-R1 model where the LLM
learns -- solely through reinforcement learning (RL) -- to autonomously
generate (multiple) search queries during step-by-step reasoning with real-time
retrieval. Search-R1 optimizes LLM rollouts with multi-turn search
interactions, leveraging retrieved token masking for stable RL training and a
simple outcome-based reward function. Experiments on seven question-answering
datasets show that Search-R1 improves performance by 26% (Qwen2.5-7B), 21%
(Qwen2.5-3B), and 10% (LLaMA3.2-3B) over SOTA baselines. This paper further
provides empirical insights into RL optimization methods, LLM choices, and
response length dynamics in retrieval-augmented reasoning. The code and model
checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.
Search-R1, an extension of DeepSeek-R1, enhances large language models by autonomously generating multiple search queries during step-by-step reasoning using reinforcement learning, improving performance in retrieval-augmented question-answering tasks.
retrieval augmentation reinforcement learning search queries step-by-step reasoning real-time retrieval outcome-based reward function multi-turn search interactions token masking