SurveyBench: How Well Can LLM(-Agents) Write Academic Surveys?
Academic survey writing, which distills vast literature into a coherent and
insightful narrative, remains a labor-intensive and intellectually demanding
task. While recent approaches, such as general DeepResearch agents and
survey-specialized methods, can generate surveys automatically (a.k.a.
LLM4Survey), their outputs often fall short of human standards and there lacks
a rigorous, reader-aligned benchmark for thoroughly revealing their
deficiencies. To fill the gap, we propose a fine-grained, quiz-driven
evaluation framework SurveyBench, featuring (1) typical survey topics source
from recent 11,343 arXiv papers and corresponding 4,947 high-quality surveys;
(2) a multifaceted metric hierarchy that assesses the outline quality (e.g.,
coverage breadth, logical coherence), content quality (e.g., synthesis
granularity, clarity of insights), and non-textual richness; and (3) a
dual-mode evaluation protocol that includes content-based and quiz-based
answerability tests, explicitly aligned with readers' informational needs.
Results show SurveyBench effectively challenges existing LLM4Survey approaches
(e.g., on average 21% lower than human in content-based evaluation).
68e322c073e20ab577841fbb
Zhaojun Sun
68e322c073e20ab577841fbc
Xuzhou Zhu
68e322c073e20ab577841fbd
Xuanhe Zhou
68e322c073e20ab577841fbe
Xin Tong
68e322c073e20ab577841fbf
Shuo Wang
68e322c073e20ab577841fc0
Jie Fu
68e322c073e20ab577841fc1
Guoliang Li
68e322c073e20ab577841fc2
Zhiyuan Liu
68e322c073e20ab577841fc3
Fan Wu
SurveyBench: How Well Can LLM(-Agents) Write Academic Surveys?
Academic survey writing, which distills vast literature into a coherent and
insightful narrative, remains a labor-intensive and intellectually demanding
task. While recent approaches, such as general DeepResearch agents and
survey-specialized methods, can generate surveys automatically (a.k.a.
LLM4Survey), their outputs often fall short of human standards and there lacks
a rigorous, reader-aligned benchmark for thoroughly revealing their
deficiencies. To fill the gap, we propose a fine-grained, quiz-driven
evaluation framework SurveyBench, featuring (1) typical survey topics source
from recent 11,343 arXiv papers and corresponding 4,947 high-quality surveys;
(2) a multifaceted metric hierarchy that assesses the outline quality (e.g.,
coverage breadth, logical coherence), content quality (e.g., synthesis
granularity, clarity of insights), and non-textual richness; and (3) a
dual-mode evaluation protocol that includes content-based and quiz-based
answerability tests, explicitly aligned with readers' informational needs.
Results show SurveyBench effectively challenges existing LLM4Survey approaches
(e.g., on average 21% lower than human in content-based evaluation).
A new evaluation framework, SurveyBench, assesses the quality of automatically generated academic surveys using a quiz-driven approach, revealing deficiencies in current LLM4Survey methods.
DeepResearch agents LLM4Survey SurveyBench arXiv papers high-quality surveys outline quality content quality synthesis granularity logical coherence non-textual richness content-based evaluation quiz-based evaluation