Graph2Eval: Automatic Multimodal Task Generation for Agents via
  Knowledge Graphs
As multimodal LLM-driven agents continue to advance in autonomy and
generalization, evaluation based on static datasets can no longer adequately
assess their true capabilities in dynamic environments and diverse tasks.
Existing LLM-based synthetic data methods are largely designed for LLM training
and evaluation, and thus cannot be directly applied to agent tasks that require
tool use and interactive capabilities. While recent studies have explored
automatic agent task generation with LLMs, most efforts remain limited to text
or image analysis, without systematically modeling multi-step interactions in
web environments. To address these challenges, we propose Graph2Eval, a
knowledge graph-based framework that automatically generates both multimodal
document comprehension tasks and web interaction tasks, enabling comprehensive
evaluation of agents' reasoning, collaboration, and interactive capabilities.
In our approach, knowledge graphs constructed from multi-source external data
serve as the task space, where we translate semantic relations into structured
multimodal tasks using subgraph sampling, task templates, and meta-paths. A
multi-stage filtering pipeline based on node reachability, LLM scoring, and
similarity analysis is applied to guarantee the quality and executability of
the generated tasks. Furthermore, Graph2Eval supports end-to-end evaluation of
multiple agent types (Single-Agent, Multi-Agent, Web Agent) and measures
reasoning, collaboration, and interaction capabilities. We instantiate the
framework with Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning
document comprehension and web interaction scenarios. Experiments show that
Graph2Eval efficiently generates tasks that differentiate agent and model
performance, revealing gaps in reasoning, collaboration, and web interaction
across different settings and offering a new perspective for agent evaluation.
68e48ad7e4e093a7044e4d8a
Yurun Chen
68e48ad7e4e093a7044e4d8b
Xavier Hu
68e48ad7e4e093a7044e4d8c
Yuhan Liu
68e48ad7e4e093a7044e4d8d
Ziqi Wang
68e48ad7e4e093a7044e4d8e
Zeyi Liao
68e48ad7e4e093a7044e4d8f
Lin Chen
68e48ad7e4e093a7044e4d90
Feng Wei
68e48ad7e4e093a7044e4d91
Yuxi Qian
68e48ad7e4e093a7044e4d92
Bo Zheng
68e48ad7e4e093a7044e4d93
Keting Yin
68e48ad7e4e093a7044e4d94
Shengyu Zhang
Graph2Eval: Automatic Multimodal Task Generation for Agents via
  Knowledge Graphs
As multimodal LLM-driven agents continue to advance in autonomy and
generalization, evaluation based on static datasets can no longer adequately
assess their true capabilities in dynamic environments and diverse tasks.
Existing LLM-based synthetic data methods are largely designed for LLM training
and evaluation, and thus cannot be directly applied to agent tasks that require
tool use and interactive capabilities. While recent studies have explored
automatic agent task generation with LLMs, most efforts remain limited to text
or image analysis, without systematically modeling multi-step interactions in
web environments. To address these challenges, we propose Graph2Eval, a
knowledge graph-based framework that automatically generates both multimodal
document comprehension tasks and web interaction tasks, enabling comprehensive
evaluation of agents' reasoning, collaboration, and interactive capabilities.
In our approach, knowledge graphs constructed from multi-source external data
serve as the task space, where we translate semantic relations into structured
multimodal tasks using subgraph sampling, task templates, and meta-paths. A
multi-stage filtering pipeline based on node reachability, LLM scoring, and
similarity analysis is applied to guarantee the quality and executability of
the generated tasks. Furthermore, Graph2Eval supports end-to-end evaluation of
multiple agent types (Single-Agent, Multi-Agent, Web Agent) and measures
reasoning, collaboration, and interaction capabilities. We instantiate the
framework with Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning
document comprehension and web interaction scenarios. Experiments show that
Graph2Eval efficiently generates tasks that differentiate agent and model
performance, revealing gaps in reasoning, collaboration, and web interaction
across different settings and offering a new perspective for agent evaluation.
Graph2Eval, a knowledge graph-based framework, generates multimodal and interactive tasks to comprehensively evaluate agents' reasoning, collaboration, and web interaction capabilities.
multimodal LLM-driven agents knowledge graph subgraph sampling task templates meta-paths node reachability LLM scoring similarity analysis Graph2Eval-Bench document comprehension web interaction tasks Single-Agent Multi-Agent Web Agent