MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large
  Vision and Language Models
Vision-and-Language Models (VLMs) have shown impressive capabilities on
single-turn benchmarks, yet real-world applications often demand more intricate
multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only
partially capture the breadth and depth of conversational scenarios encountered
by users. In this work, we introduce MultiVerse, a novel multi-turn
conversation benchmark featuring 647 dialogues - each averaging four turns -
derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484
tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from
factual knowledge and perception to advanced reasoning tasks such as
mathematics and coding. To facilitate robust assessment, we propose a
checklist-based evaluation method that leverages GPT-4o as the automated
evaluator, measuring performance across 37 key aspects, including perceptual
accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on
MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve
only a 50% success rate in complex multi-turn conversations, highlighting the
dataset's challenging nature. Notably, we find that providing full dialogue
context significantly enhances performance for smaller or weaker models,
emphasizing the importance of in-context learning. We believe MultiVerse is a
landscape of evaluating multi-turn interaction abilities for VLMs.
68f7390224c4489363111b4b
Young-Jun Lee
68f7390224c4489363111b4c
Byung-Kwan Lee
68f7390224c4489363111b4d
Jianshu Zhang
68f7390224c4489363111b4e
Yechan Hwang
68f7390224c4489363111b4f
Byungsoo Ko
68f7390224c4489363111b50
Han-Gyu Kim
68f7390224c4489363111b51
Dongyu Yao
68f7390224c4489363111b52
Xuankun Rong
68f7390224c4489363111b53
Eojin Joo
68f7390224c4489363111b54
Seung-Ho Han
68f7390224c4489363111b55
Bowon Ko
68f7390224c4489363111b56
Ho-Jin Choi
MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large
  Vision and Language Models
Vision-and-Language Models (VLMs) have shown impressive capabilities on
single-turn benchmarks, yet real-world applications often demand more intricate
multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only
partially capture the breadth and depth of conversational scenarios encountered
by users. In this work, we introduce MultiVerse, a novel multi-turn
conversation benchmark featuring 647 dialogues - each averaging four turns -
derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484
tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from
factual knowledge and perception to advanced reasoning tasks such as
mathematics and coding. To facilitate robust assessment, we propose a
checklist-based evaluation method that leverages GPT-4o as the automated
evaluator, measuring performance across 37 key aspects, including perceptual
accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on
MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve
only a 50% success rate in complex multi-turn conversations, highlighting the
dataset's challenging nature. Notably, we find that providing full dialogue
context significantly enhances performance for smaller or weaker models,
emphasizing the importance of in-context learning. We believe MultiVerse is a
landscape of evaluating multi-turn interaction abilities for VLMs.
MultiVerse, a new multi-turn conversation benchmark, evaluates VLMs across diverse tasks and interaction goals, revealing challenges and the importance of in-context learning.
Vision-and-Language Models multi-turn dialogues MMDU ConvBench MultiVerse GPT-4o checklist-based evaluation perceptual accuracy linguistic clarity factual correctness in-context learning