OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real
  Computer Environments
Autonomous agents that accomplish complex computer tasks with minimal human
interventions have the potential to transform human-computer interaction,
significantly enhancing accessibility and productivity. However, existing
benchmarks either lack an interactive environment or are limited to
environments specific to certain applications or domains, failing to reflect
the diverse and complex nature of real-world computer use, thereby limiting the
scope of tasks and agent scalability. To address this issue, we introduce
OSWorld, the first-of-its-kind scalable, real computer environment for
multimodal agents, supporting task setup, execution-based evaluation, and
interactive learning across various operating systems such as Ubuntu, Windows,
and macOS. OSWorld can serve as a unified, integrated computer environment for
assessing open-ended computer tasks that involve arbitrary applications.
Building upon OSWorld, we create a benchmark of 369 computer tasks involving
real web and desktop apps in open domains, OS file I/O, and workflows spanning
multiple applications. Each task example is derived from real-world computer
use cases and includes a detailed initial state setup configuration and a
custom execution-based evaluation script for reliable, reproducible evaluation.
Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
reveals significant deficiencies in their ability to serve as computer
assistants. While humans can accomplish over 72.36% of the tasks, the best
model achieves only 12.24% success, primarily struggling with GUI grounding and
operational knowledge. Comprehensive analysis using OSWorld provides valuable
insights for developing multimodal generalist agents that were not possible
with previous benchmarks. Our code, environment, baseline models, and data are
publicly available at https://os-world.github.io.
66188c6be1abd2f76dab9661
Tianbao Xie
claimed_verified
2024-04-12T03:05:11.439Z
618767e4238063b4615d042b
https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
Tianbao Xie
user
tianbaoxiexxx
66188c6be1abd2f76dab9662
Danyang Zhang
claimed_verified
2024-04-12T14:49:49.941Z
6455ebad654d8bccae518aaf
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qTvoGqVBuQavfcgbwIoQq.jpeg
Danyang Zhang
user
zdy023
66188c6be1abd2f76dab9663
Jixuan Chen
claimed_verified
2024-04-18T13:15:48.663Z
6465941d0e6c7618f615675b
https://cdn-avatars.huggingface.co/v1/production/uploads/6465941d0e6c7618f615675b/W4EHqlCucz_bojFLFEeV_.jpeg
Jixuan Chen
user
Mayome
66188c6be1abd2f76dab9664
Xiaochuan Li
claimed_verified
2024-04-12T14:50:14.081Z
64b103cf372d434077206750
/avatars/ba0eb4fc712a8b9b93ceb30d11859ec2.svg
Xiaochuan Li
user
lixiaochuan2020
66188c6be1abd2f76dab9665
Siheng Zhao
66188c6be1abd2f76dab9666
Ruisheng Cao
claimed_verified
2024-04-12T14:49:54.777Z
5fbcf0d28f35b82700205fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/5fbcf0d28f35b82700205fd7/KZPwcaIymBAPs0UHXe5fa.jpeg
Ruisheng Cao
user
rshcao
66188c6be1abd2f76dab9667
Toh Jing Hua
claimed_verified
2024-04-12T07:54:16.193Z
64871af469f5c89076c4d1ef
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/a2x213gR2JSFfzxWyHklT.jpeg
Toh Jing Hua
user
jing-hua
66188c6be1abd2f76dab9668
Zhoujun Cheng
66188c6be1abd2f76dab9669
Dongchan Shin
claimed_verified
2025-04-11T07:26:26.991Z
63340b24f68a3fb7efa62b3a
/avatars/44c960437c037553d90b1ca24c952977.svg
Dongchan Shin
user
ShinDC
66188c6be1abd2f76dab966a
Fangyu Lei
claimed_verified
2025-03-18T14:58:47.418Z
64104b467a15af878ae6695d
/avatars/407983918c12411e5ed636bf7435522b.svg
Fangyu Lei
user
FangyuLei
66188c6be1abd2f76dab966b
Yitao Liu
66188c6be1abd2f76dab966c
Yiheng Xu
claimed_verified
2024-11-03T14:25:25.357Z
601d29ab913ad3afd7b7ddb8
https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
Yiheng Xu
user
ranpox
66188c6be1abd2f76dab966d
Shuyan Zhou
66188c6be1abd2f76dab966e
Silvio Savarese
66188c6be1abd2f76dab966f
Caiming Xiong
66188c6be1abd2f76dab9670
Victor Zhong
66188c6be1abd2f76dab9671
Tao Yu
claimed_verified
2024-04-28T19:30:43.725Z
629b4c93ba19b79d354270f0
/avatars/06b8aae4ce030f4e2036d68c08cbcf39.svg
Tao Yu
user
taoyds
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real
  Computer Environments
Autonomous agents that accomplish complex computer tasks with minimal human
interventions have the potential to transform human-computer interaction,
significantly enhancing accessibility and productivity. However, existing
benchmarks either lack an interactive environment or are limited to
environments specific to certain applications or domains, failing to reflect
the diverse and complex nature of real-world computer use, thereby limiting the
scope of tasks and agent scalability. To address this issue, we introduce
OSWorld, the first-of-its-kind scalable, real computer environment for
multimodal agents, supporting task setup, execution-based evaluation, and
interactive learning across various operating systems such as Ubuntu, Windows,
and macOS. OSWorld can serve as a unified, integrated computer environment for
assessing open-ended computer tasks that involve arbitrary applications.
Building upon OSWorld, we create a benchmark of 369 computer tasks involving
real web and desktop apps in open domains, OS file I/O, and workflows spanning
multiple applications. Each task example is derived from real-world computer
use cases and includes a detailed initial state setup configuration and a
custom execution-based evaluation script for reliable, reproducible evaluation.
Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
reveals significant deficiencies in their ability to serve as computer
assistants. While humans can accomplish over 72.36% of the tasks, the best
model achieves only 12.24% success, primarily struggling with GUI grounding and
operational knowledge. Comprehensive analysis using OSWorld provides valuable
insights for developing multimodal generalist agents that were not possible
with previous benchmarks. Our code, environment, baseline models, and data are
publicly available at https://os-world.github.io.
OSWorld provides a scalable, real computer environment for evaluating multimodal agents across various operating systems and tasks, highlighting significant gaps in current LLM/VLM capabilities.
OSWorld multimodal agents interactive learning GUI grounding execution-based evaluation benchmark LLM VLM operational knowledge