Chain-of-Retrieval Augmented Generation
This paper introduces an approach for training o1-like RAG models that
retrieve and reason over relevant information step by step before generating
the final answer. Conventional RAG methods usually perform a single retrieval
step before the generation process, which limits their effectiveness in
addressing complex queries due to imperfect retrieval results. In contrast, our
proposed method, CoRAG (Chain-of-Retrieval Augmented Generation), allows the
model to dynamically reformulate the query based on the evolving state. To
train CoRAG effectively, we utilize rejection sampling to automatically
generate intermediate retrieval chains, thereby augmenting existing RAG
datasets that only provide the correct final answer. At test time, we propose
various decoding strategies to scale the model's test-time compute by
controlling the length and number of sampled retrieval chains. Experimental
results across multiple benchmarks validate the efficacy of CoRAG, particularly
in multi-hop question answering tasks, where we observe more than 10 points
improvement in EM score compared to strong baselines. On the KILT benchmark,
CoRAG establishes a new state-of-the-art performance across a diverse range of
knowledge-intensive tasks. Furthermore, we offer comprehensive analyses to
understand the scaling behavior of CoRAG, laying the groundwork for future
research aimed at developing factual and grounded foundation models.
6796fa2d4fccd4b951e8ab0a
Liang Wang
6796fa2d4fccd4b951e8ab0b
Haonan Chen
claimed_verified
2025-02-13T08:26:16.425Z
66add675c7a575aa0e03d5f3
/avatars/b72b18130664c1de197c1f8df371aa70.svg
Haonan Chen
user
Haon-Chen
6796fa2d4fccd4b951e8ab0c
Nan Yang
6796fa2d4fccd4b951e8ab0d
Xiaolong Huang
admin_assigned
2025-01-27T14:59:51.272Z
64149ba8e682c73521d46ecb
/avatars/c1859aa2d295077e798c8bdc83664a07.svg
Xiaolong Huang
user
DiamondH
6796fa2d4fccd4b951e8ab0e
Zhicheng Dou
admin_assigned
2025-01-27T14:59:36.119Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
6796fa2d4fccd4b951e8ab0f
Furu Wei
admin_assigned
2025-01-27T14:59:30.095Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Chain-of-Retrieval Augmented Generation
This paper introduces an approach for training o1-like RAG models that
retrieve and reason over relevant information step by step before generating
the final answer. Conventional RAG methods usually perform a single retrieval
step before the generation process, which limits their effectiveness in
addressing complex queries due to imperfect retrieval results. In contrast, our
proposed method, CoRAG (Chain-of-Retrieval Augmented Generation), allows the
model to dynamically reformulate the query based on the evolving state. To
train CoRAG effectively, we utilize rejection sampling to automatically
generate intermediate retrieval chains, thereby augmenting existing RAG
datasets that only provide the correct final answer. At test time, we propose
various decoding strategies to scale the model's test-time compute by
controlling the length and number of sampled retrieval chains. Experimental
results across multiple benchmarks validate the efficacy of CoRAG, particularly
in multi-hop question answering tasks, where we observe more than 10 points
improvement in EM score compared to strong baselines. On the KILT benchmark,
CoRAG establishes a new state-of-the-art performance across a diverse range of
knowledge-intensive tasks. Furthermore, we offer comprehensive analyses to
understand the scaling behavior of CoRAG, laying the groundwork for future
research aimed at developing factual and grounded foundation models.
CoRAG, a multi-step retrieval and reasoning approach, enhances RAG models by dynamically refining queries and using rejection sampling to improve performance, especially in multi-hop question answering.
RAG models CoRAG rejection sampling intermediate retrieval chains decoding strategies EM score KILT benchmark multi-hop question answering