Spider2-V: How Far Are Multimodal Agents From Automating Data Science
  and Engineering Workflows?
Data science and engineering workflows often span multiple stages, from
warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As
vision language models (VLMs) advance in multimodal understanding and code
generation, VLM-based agents could potentially automate these workflows by
generating SQL queries, Python code, and GUI operations. This automation can
improve the productivity of experts while democratizing access to large-scale
data analysis. In this paper, we introduce Spider2-V, the first multimodal
agent benchmark focusing on professional data science and engineering
workflows, featuring 494 real-world tasks in authentic computer environments
and incorporating 20 enterprise-level professional applications. These tasks,
derived from real-world use cases, evaluate the ability of a multimodal agent
to perform data-related tasks by writing code and managing the GUI in
enterprise data software systems. To balance realistic simulation with
evaluation simplicity, we devote significant effort to developing automatic
configurations for task setup and carefully crafting evaluation metrics for
each task. Furthermore, we supplement multimodal agents with comprehensive
documents of these enterprise data software systems. Our empirical evaluation
reveals that existing state-of-the-art LLM/VLM-based agents do not reliably
automate full data workflows (14.0% success). Even with step-by-step guidance,
these agents still underperform in tasks that require fine-grained,
knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted
workspaces (10.6%). We hope that Spider2-V paves the way for autonomous
multimodal agents to transform the automation of data science and engineering
workflow. Our code and data are available at https://spider2-v.github.io.
6696c1d3f97e6a8a930f2080
Ruisheng Cao
claimed_verified
2024-07-23T07:48:32.490Z
5fbcf0d28f35b82700205fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/5fbcf0d28f35b82700205fd7/KZPwcaIymBAPs0UHXe5fa.jpeg
Ruisheng Cao
user
rshcao
6696c1d3f97e6a8a930f2081
Fangyu Lei
claimed_verified
2025-03-18T14:58:26.657Z
64104b467a15af878ae6695d
/avatars/407983918c12411e5ed636bf7435522b.svg
Fangyu Lei
user
FangyuLei
6696c1d3f97e6a8a930f2082
Haoyuan Wu
claimed_verified
2025-05-20T19:30:05.532Z
653927c90dfecb4b26bc5885
https://cdn-avatars.huggingface.co/v1/production/uploads/653927c90dfecb4b26bc5885/lM7d75M0CzVrb6HVz1dDD.png
Fred Wu
user
HappySixHF
6696c1d3f97e6a8a930f2083
Jixuan Chen
claimed_verified
2024-09-03T14:49:04.929Z
6465941d0e6c7618f615675b
https://cdn-avatars.huggingface.co/v1/production/uploads/6465941d0e6c7618f615675b/W4EHqlCucz_bojFLFEeV_.jpeg
Jixuan Chen
user
Mayome
6696c1d3f97e6a8a930f2084
Yeqiao Fu
6696c1d3f97e6a8a930f2085
Hongcheng Gao
claimed_verified
2024-07-18T09:07:52.780Z
62728f4f6253fe2068da1021
https://cdn-avatars.huggingface.co/v1/production/uploads/62728f4f6253fe2068da1021/KZ65X0EH98AF3zXemPiap.jpeg
Hongcheng Gao
user
HongchengGao
6696c1d3f97e6a8a930f2086
Xinzhuang Xiong
6696c1d3f97e6a8a930f2087
Hanchong Zhang
6696c1d3f97e6a8a930f2088
Yuchen Mao
6696c1d3f97e6a8a930f2089
Wenjing Hu
6696c1d3f97e6a8a930f208a
Tianbao Xie
claimed_verified
2024-07-19T11:23:53.142Z
618767e4238063b4615d042b
https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
Tianbao Xie
user
tianbaoxiexxx
6696c1d3f97e6a8a930f208b
Hongshen Xu
6696c1d3f97e6a8a930f208c
Danyang Zhang
6696c1d3f97e6a8a930f208d
Sida Wang
6696c1d3f97e6a8a930f208e
Ruoxi Sun
6696c1d3f97e6a8a930f208f
Pengcheng Yin
6696c1d3f97e6a8a930f2090
Caiming Xiong
6696c1d3f97e6a8a930f2091
Ansong Ni
6696c1d3f97e6a8a930f2092
Qian Liu
claimed_verified
2024-11-15T09:46:23.223Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
6696c1d3f97e6a8a930f2093
Victor Zhong
6696c1d3f97e6a8a930f2094
Lu Chen
6696c1d3f97e6a8a930f2095
Kai Yu
6696c1d3f97e6a8a930f2096
Tao Yu
Spider2-V: How Far Are Multimodal Agents From Automating Data Science
  and Engineering Workflows?
Data science and engineering workflows often span multiple stages, from
warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As
vision language models (VLMs) advance in multimodal understanding and code
generation, VLM-based agents could potentially automate these workflows by
generating SQL queries, Python code, and GUI operations. This automation can
improve the productivity of experts while democratizing access to large-scale
data analysis. In this paper, we introduce Spider2-V, the first multimodal
agent benchmark focusing on professional data science and engineering
workflows, featuring 494 real-world tasks in authentic computer environments
and incorporating 20 enterprise-level professional applications. These tasks,
derived from real-world use cases, evaluate the ability of a multimodal agent
to perform data-related tasks by writing code and managing the GUI in
enterprise data software systems. To balance realistic simulation with
evaluation simplicity, we devote significant effort to developing automatic
configurations for task setup and carefully crafting evaluation metrics for
each task. Furthermore, we supplement multimodal agents with comprehensive
documents of these enterprise data software systems. Our empirical evaluation
reveals that existing state-of-the-art LLM/VLM-based agents do not reliably
automate full data workflows (14.0% success). Even with step-by-step guidance,
these agents still underperform in tasks that require fine-grained,
knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted
workspaces (10.6%). We hope that Spider2-V paves the way for autonomous
multimodal agents to transform the automation of data science and engineering
workflow. Our code and data are available at https://spider2-v.github.io.
Spider2-V, a multimodal agent benchmark, evaluates the automation of professional data science workflows using SQL, Python, and GUI interactions, revealing limitations in existing state-of-the-art LLM/VLM-based agents.
vision language models multimodal understanding multimodal agent SQL queries Python code GUI operations data science engineering workflows enterprise data software systems evaluation metrics