Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning
  of LLMs
Large language models have demonstrated impressive reasoning capabilities but
are inherently limited by their knowledge reservoir. Retrieval-augmented
reasoning mitigates this limitation by allowing LLMs to query external
resources, but existing methods often retrieve irrelevant or noisy information,
hindering accurate reasoning. In this paper, we propose AutoRefine, a
reinforcement learning post-training framework that adopts a new
``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit
knowledge refinement steps between successive search calls, enabling the model
to iteratively filter, distill, and organize evidence before generating an
answer. Furthermore, we incorporate tailored retrieval-specific rewards
alongside answer correctness rewards using group relative policy optimization.
Experiments on single-hop and multi-hop QA benchmarks demonstrate that
AutoRefine significantly outperforms existing approaches, particularly in
complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine
issues frequent, higher-quality searches and synthesizes evidence effectively.
6834f8d4bb7d1147551cc8f6
Yaorui Shi
claimed_verified
2025-05-28T09:00:51.919Z
63edd2d1f765928ceeb49057
https://cdn-avatars.huggingface.co/v1/production/uploads/1676530369930-noauth.png
Yaorui SHI
user
yrshi
6834f8d4bb7d1147551cc8f7
Sihang Li
6834f8d4bb7d1147551cc8f8
Chang Wu
6834f8d4bb7d1147551cc8f9
Zhiyuan Liu
claimed_verified
2025-06-02T07:49:10.763Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
6834f8d4bb7d1147551cc8fa
Junfeng Fang
6834f8d4bb7d1147551cc8fb
Hengxing Cai
admin_assigned
2025-05-28T12:35:49.890Z
6340c214181f5648a58f5f82
/avatars/2ee353d8ec2b0ebc5b58d0641beddc26.svg
Hengxing Cai
user
enjoy
6834f8d4bb7d1147551cc8fc
An Zhang
6834f8d4bb7d1147551cc8fd
Xiang Wang
Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning
  of LLMs
Large language models have demonstrated impressive reasoning capabilities but
are inherently limited by their knowledge reservoir. Retrieval-augmented
reasoning mitigates this limitation by allowing LLMs to query external
resources, but existing methods often retrieve irrelevant or noisy information,
hindering accurate reasoning. In this paper, we propose AutoRefine, a
reinforcement learning post-training framework that adopts a new
``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit
knowledge refinement steps between successive search calls, enabling the model
to iteratively filter, distill, and organize evidence before generating an
answer. Furthermore, we incorporate tailored retrieval-specific rewards
alongside answer correctness rewards using group relative policy optimization.
Experiments on single-hop and multi-hop QA benchmarks demonstrate that
AutoRefine significantly outperforms existing approaches, particularly in
complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine
issues frequent, higher-quality searches and synthesizes evidence effectively.
AutoRefine, a reinforcement learning framework for large language models, enhances retrieval-augmented reasoning by iteratively refining knowledge and optimizing searches, leading to improved performance in complex question-answering tasks.
Large language models Retrieval-augmented reasoning AutoRefine Reinforcement learning Search-and-refine-during-think Reinforcement learning post-training Group relative policy optimization Single-hop QA benchmarks Multi-hop QA benchmarks