Search Arena: Analyzing Search-Augmented LLMs
Search-augmented language models combine web search with Large Language
Models (LLMs) to improve response groundedness and freshness. However,
analyzing these systems remains challenging: existing datasets are limited in
scale and narrow in scope, often constrained to static, single-turn,
fact-checking questions. In this work, we introduce Search Arena, a
crowd-sourced, large-scale, human-preference dataset of over 24,000 paired
multi-turn user interactions with search-augmented LLMs. The dataset spans
diverse intents and languages, and contains full system traces with around
12,000 human preference votes. Our analysis reveals that user preferences are
influenced by the number of citations, even when the cited content does not
directly support the attributed claims, uncovering a gap between perceived and
actual credibility. Furthermore, user preferences vary across cited sources,
revealing that community-driven platforms are generally preferred and static
encyclopedic sources are not always appropriate and reliable. To assess
performance across different settings, we conduct cross-arena analyses by
testing search-augmented LLMs in a general-purpose chat environment and
conventional LLMs in search-intensive settings. We find that web search does
not degrade and may even improve performance in non-search settings; however,
the quality in search settings is significantly affected if solely relying on
the model's parametric knowledge. We open-sourced the dataset to support future
research in this direction. Our dataset and code are available at:
https://github.com/lmarena/search-arena.
68430c258f9ec8394c514870
Mihran Miroyan
claimed_verified
2025-07-22T07:54:23.502Z
647a99bd61e1252d761ae6ed
https://cdn-avatars.huggingface.co/v1/production/uploads/647a99bd61e1252d761ae6ed/CI67hb26bP9l1yIA_NIGE.jpeg
Mir Miroyan
user
mmiroyan
68430c258f9ec8394c514871
Tsung-Han Wu
claimed_verified
2025-06-07T05:45:14.475Z
644a767044b75fd95805d232
https://cdn-avatars.huggingface.co/v1/production/uploads/644a767044b75fd95805d232/vHA2vI_B3CpXapdBEwspB.jpeg
Patrick (Tsung-Han) Wu
user
tsunghanwu
68430c258f9ec8394c514872
Logan King
68430c258f9ec8394c514873
Tianle Li
68430c258f9ec8394c514874
Jiayi Pan
68430c258f9ec8394c514875
Xinyan Hu
68430c258f9ec8394c514876
Wei-Lin Chiang
68430c258f9ec8394c514877
Anastasios N. Angelopoulos
68430c258f9ec8394c514878
Trevor Darrell
68430c258f9ec8394c514879
Narges Norouzi
68430c258f9ec8394c51487a
Joseph E. Gonzalez
Search Arena: Analyzing Search-Augmented LLMs
Search-augmented language models combine web search with Large Language
Models (LLMs) to improve response groundedness and freshness. However,
analyzing these systems remains challenging: existing datasets are limited in
scale and narrow in scope, often constrained to static, single-turn,
fact-checking questions. In this work, we introduce Search Arena, a
crowd-sourced, large-scale, human-preference dataset of over 24,000 paired
multi-turn user interactions with search-augmented LLMs. The dataset spans
diverse intents and languages, and contains full system traces with around
12,000 human preference votes. Our analysis reveals that user preferences are
influenced by the number of citations, even when the cited content does not
directly support the attributed claims, uncovering a gap between perceived and
actual credibility. Furthermore, user preferences vary across cited sources,
revealing that community-driven platforms are generally preferred and static
encyclopedic sources are not always appropriate and reliable. To assess
performance across different settings, we conduct cross-arena analyses by
testing search-augmented LLMs in a general-purpose chat environment and
conventional LLMs in search-intensive settings. We find that web search does
not degrade and may even improve performance in non-search settings; however,
the quality in search settings is significantly affected if solely relying on
the model's parametric knowledge. We open-sourced the dataset to support future
research in this direction. Our dataset and code are available at:
https://github.com/lmarena/search-arena.
Search Arena is a large-scale human-preference dataset that analyzes user interactions with search-augmented language models, revealing insights into citation influence and source credibility.
LLMs search-augmented language models dataset human-preference user interactions citations credibility community-driven platforms search-intensive settings parametric knowledge