AceRAG: Advancing Reasoning-Intensive Retrieval-Augmented Generation via LLM Self-Play
Ran Xu, Yuchen Zhuang, Zihan Dong, Ruiyu Wang, Yue Yu, Joyce Ho, Linjun Zhang, Haoyu Wang, Wenqi Shi, Carl Yang
Retrieval-augmented generation (RAG) systems often struggle with complex reasoning tasks due to ineffective multi-hop retrieval and limited reasoning ability. We propose AceRAG, a cooperative self-play framework that trains a single large language model (LLM) to alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation. AceRAG couples supervised fine-tuning on a diverse mixture of retrieval, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations. Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceRAG outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level reasoning tasks, AceRAG-32B matches the performance of the giant DeepSeek-V3 model using less than 5% of its parameters. Even at smaller scales (1.5B and 8B), AceRAG often surpasses existing RAG models with up to 9x more parameters, highlighting its exceptional efficiency and effectiveness in tackling complex reasoning tasks.