Benchmarking Agentic Workflow Generation
Large Language Models (LLMs), with their exceptional ability to handle a wide
range of tasks, have driven significant advancements in tackling reasoning and
planning tasks, wherein decomposing complex problems into executable workflows
is a crucial step in this process. Existing workflow evaluation frameworks
either focus solely on holistic performance or suffer from limitations such as
restricted scenario coverage, simplistic workflow structures, and lax
evaluation standards. To this end, we introduce WorFBench, a unified workflow
generation benchmark with multi-faceted scenarios and intricate graph workflow
structures. Additionally, we present WorFEval, a systemic evaluation protocol
utilizing subsequence and subgraph matching algorithms to accurately quantify
the LLM agent's workflow generation capabilities. Through comprehensive
evaluations across different types of LLMs, we discover distinct gaps between
the sequence planning capabilities and graph planning capabilities of LLM
agents, with even GPT-4 exhibiting a gap of around 15%. We also train two
open-source models and evaluate their generalization abilities on held-out
tasks. Furthermore, we observe that the generated workflows can enhance
downstream tasks, enabling them to achieve superior performance with less time
during inference. Code and dataset will be available at
https://github.com/zjunlp/WorFBench.
6708dea5d5c3f877d9be613d
Shuofei Qiao
admin_assigned
2024-10-11T10:33:25.748Z
6447800f30fa4ecb85ddad80
https://cdn-avatars.huggingface.co/v1/production/uploads/6447800f30fa4ecb85ddad80/NsmXIaMsWctmTNA7tFVkX.jpeg
Shuofei Qiao
user
GoooDte
6708dea5d5c3f877d9be613e
Runnan Fang
admin_assigned
2024-10-11T10:33:31.931Z
63d32cd7b734eaa4d4fa410b
/avatars/68acb80f62bc6493e1ad26506999b6c4.svg
Runnan Fang
user
Runnaning
6708dea5d5c3f877d9be613f
Zhisong Qiu
admin_assigned
2024-10-11T10:33:37.295Z
65535b54140fc44a74d43635
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MIrD8OzDKF2aI38i7ZPjR.jpeg
Zhisong Qiu
user
consultantQ
6708dea5d5c3f877d9be6140
Xiaobin Wang
admin_assigned
2024-10-11T10:33:42.698Z
654e24cb12c9597b2099385e
/avatars/c7505e894fe0b5e3fc1db86d5717da8f.svg
XIAOBIN WANG
user
xiaoyuehanbin
6708dea5d5c3f877d9be6141
Ningyu Zhang
claimed_verified
2024-10-11T10:08:42.030Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
6708dea5d5c3f877d9be6142
Yong Jiang
claimed_verified
2025-01-20T09:31:08.525Z
678ddd806aa55e76bfffb953
/avatars/f447936c286a6a2d2874a760210b2f17.svg
Yong Jiang
user
yongjiangNLP
6708dea5d5c3f877d9be6143
Pengjun Xie
admin_assigned
2024-10-11T10:33:56.901Z
63a091e42fabbbb89991f5ce
/avatars/d55485b06461764c36c9edf9d6e8892c.svg
pengjun xie
user
xpjandy
6708dea5d5c3f877d9be6144
Fei Huang
6708dea5d5c3f877d9be6145
Huajun Chen
admin_assigned
2024-10-11T10:34:31.686Z
64931296137833d7ec7689cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
Huajun Chen
user
huajunsir
Benchmarking Agentic Workflow Generation
Large Language Models (LLMs), with their exceptional ability to handle a wide
range of tasks, have driven significant advancements in tackling reasoning and
planning tasks, wherein decomposing complex problems into executable workflows
is a crucial step in this process. Existing workflow evaluation frameworks
either focus solely on holistic performance or suffer from limitations such as
restricted scenario coverage, simplistic workflow structures, and lax
evaluation standards. To this end, we introduce WorFBench, a unified workflow
generation benchmark with multi-faceted scenarios and intricate graph workflow
structures. Additionally, we present WorFEval, a systemic evaluation protocol
utilizing subsequence and subgraph matching algorithms to accurately quantify
the LLM agent's workflow generation capabilities. Through comprehensive
evaluations across different types of LLMs, we discover distinct gaps between
the sequence planning capabilities and graph planning capabilities of LLM
agents, with even GPT-4 exhibiting a gap of around 15%. We also train two
open-source models and evaluate their generalization abilities on held-out
tasks. Furthermore, we observe that the generated workflows can enhance
downstream tasks, enabling them to achieve superior performance with less time
during inference. Code and dataset will be available at
https://github.com/zjunlp/WorFBench.
WorFBench and WorFEval create a comprehensive framework for evaluating large language models' workflow generation, uncovering gaps in sequence and graph planning and demonstrating improved performance in downstream tasks.
Large Language Models (LLMs) WorFBench WorFEval subsequence matching subgraph matching workflow generation graph planning sequence planning