MS-Bench: A Multimodal Benchmark for Evaluating Large Language Models in Ancient ManuScript Study
Yuqing Zhang, Yue Han, Shuanghe Zhu, Haoxiang Wu, Hangqi Li, Shengyu Zhang, Junchi Yan, Zemin Liu, Kun Kuang, Huaiyong Dou, Yongquan Zhang, Fei Wu
Analyzing ancient manuscripts has traditionally been a labor-intensive and time-consuming task for philologists. Recent advancements in LMMs have demonstrated their potential across various domains, yet their effectiveness in manuscript study remains largely unexplored. In this paper, we present MS-Bench, the first comprehensive benchmark co-developed with archaeologists, comprising 5,076 high-resolution images from 4th to 14th century and 9,982 expert-curated questions across nine sub-tasks aligned with archaeological workflows. Through four prompting strategies, we systematically evaluate 32 LMMs on their effectiveness, robustness, and cultural contextualization. Our analysis reveals scale-driven performance and reliability improvements, prompting strategies' impact on performance (CoT has two-sides effect, while Retrieval-augmented Visual Context provide consistent boost), and task preferences based on LMM's visual capabilities. Although current LMMs are not yet capable of replacing domain expertise, they show promising potential to accelerate manuscript research through future human-AI collaboration.