RAVine: Reality-Aligned Evaluation for Agentic Search
Agentic search, as a more autonomous and adaptive paradigm of retrieval
augmentation, is driving the evolution of intelligent search systems. However,
existing evaluation frameworks fail to align well with the goals of agentic
search. First, the complex queries commonly used in current benchmarks often
deviate from realistic user search scenarios. Second, prior approaches tend to
introduce noise when extracting ground truth for end-to-end evaluations,
leading to distorted assessments at a fine-grained level. Third, most current
frameworks focus solely on the quality of final answers, neglecting the
evaluation of the iterative process inherent to agentic search. To address
these limitations, we propose RAVine -- a Reality-Aligned eValuation framework
for agentic LLMs with search. RAVine targets multi-point queries and long-form
answers that better reflect user intents, and introduces an attributable ground
truth construction strategy to enhance the accuracy of fine-grained evaluation.
Moreover, RAVine examines model's interaction with search tools throughout the
iterative process, and accounts for factors of efficiency. We benchmark a
series of models using RAVine and derive several insights, which we hope will
contribute to advancing the development of agentic search systems. The code and
datasets are available at https://github.com/SwordFaith/RAVine.
68807c5900f4b5a05f2fbeab
Yilong Xu
admin_assigned
2025-07-24T12:08:21.488Z
643a587fe2b979ae6141b193
/avatars/1726b6a1629d800795f9bdf6d03ad190.svg
yilong xu
user
sapphirex
68807c5900f4b5a05f2fbeac
Xiang Long
68807c5900f4b5a05f2fbead
Zhi Zheng
68807c5900f4b5a05f2fbeae
Jinhua Gao
RAVine: Reality-Aligned Evaluation for Agentic Search
Agentic search, as a more autonomous and adaptive paradigm of retrieval
augmentation, is driving the evolution of intelligent search systems. However,
existing evaluation frameworks fail to align well with the goals of agentic
search. First, the complex queries commonly used in current benchmarks often
deviate from realistic user search scenarios. Second, prior approaches tend to
introduce noise when extracting ground truth for end-to-end evaluations,
leading to distorted assessments at a fine-grained level. Third, most current
frameworks focus solely on the quality of final answers, neglecting the
evaluation of the iterative process inherent to agentic search. To address
these limitations, we propose RAVine -- a Reality-Aligned eValuation framework
for agentic LLMs with search. RAVine targets multi-point queries and long-form
answers that better reflect user intents, and introduces an attributable ground
truth construction strategy to enhance the accuracy of fine-grained evaluation.
Moreover, RAVine examines model's interaction with search tools throughout the
iterative process, and accounts for factors of efficiency. We benchmark a
series of models using RAVine and derive several insights, which we hope will
contribute to advancing the development of agentic search systems. The code and
datasets are available at https://github.com/SwordFaith/RAVine.
RAVine is a new evaluation framework for agentic LLMs with search, focusing on realistic queries, accurate ground truth, and iterative process efficiency.
agentic search retrieval augmentation intelligent search systems evaluation frameworks complex queries ground truth end-to-end evaluations final answers iterative process Reality-Aligned eValuation framework multi-point queries long-form answers user intents attributable ground truth search tools efficiency