Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Large language models (LLMs) show remarkable potential to act as computer
agents, enhancing human productivity and software accessibility in multi-modal
tasks that require planning and reasoning. However, measuring agent performance
in realistic environments remains a challenge since: (i) most benchmarks are
limited to specific modalities or domains (e.g. text-only, web navigation, Q&A,
coding) and (ii) full benchmark evaluations are slow (on order of magnitude of
days) given the multi-step sequential nature of tasks. To address these
challenges, we introduce the Windows Agent Arena: a reproducible, general
environment focusing exclusively on the Windows operating system (OS) where
agents can operate freely within a real Windows OS and use the same wide range
of applications, tools, and web browsers available to human users when solving
tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse
Windows tasks across representative domains that require agent abilities in
planning, screen understanding, and tool usage. Our benchmark is scalable and
can be seamlessly parallelized in Azure for a full benchmark evaluation in as
little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we
also introduce a new multi-modal agent, Navi. Our agent achieves a success rate
of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted
human. Navi also demonstrates strong performance on another popular web-based
benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis
of Navi's performance, and provide insights into the opportunities for future
research in agent development and data generation using Windows Agent Arena.
  Webpage: https://microsoft.github.io/WindowsAgentArena
  Code: https://github.com/microsoft/WindowsAgentArena
66e3ac6869d1083dd958c118
Rogerio Bonatti
admin_assigned
2024-09-13T07:51:21.211Z
6600b688871b36bf846dd188
/avatars/8cc5cc42819c8dcacee10ca20ec6c057.svg
Rogerio Bonatti
user
rbonatti
66e3ac6869d1083dd958c119
Dan Zhao
66e3ac6869d1083dd958c11a
Francesco Bonacci
admin_assigned
2024-09-13T07:52:25.837Z
65ef2edf6e38bf2ff90af403
/avatars/d05249606f668d33e1b2a0fc8d676d5d.svg
Francesco Bonacci
user
francedot
66e3ac6869d1083dd958c11b
Dillon Dupont
66e3ac6869d1083dd958c11c
Sara Abdali
admin_assigned
2024-09-13T07:52:48.796Z
63eebffdbe5d5900b851bd49
https://cdn-avatars.huggingface.co/v1/production/uploads/63eebffdbe5d5900b851bd49/MIrZm3rW9nL2MwrRaDZjK.jpeg
Sara Abdali
user
Saraabdali
66e3ac6869d1083dd958c11d
Yinheng Li
admin_assigned
2024-09-13T07:52:56.140Z
622a8544305a46df46e9ba7c
/avatars/b052b3f50822a4a8f3f86ceea4bffa8e.svg
Yinheng Li
user
Inhenn
66e3ac6869d1083dd958c11e
Justin Wagle
admin_assigned
2024-09-13T07:53:03.505Z
6362a1344d4a190d6b14b7b4
/avatars/fa39fbcfff6ae16ddb0a7144463bfe83.svg
Justin Wagle
user
hrz
66e3ac6869d1083dd958c11f
Kazuhito Koishida
admin_assigned
2024-09-13T07:53:10.281Z
633f6bb8ac3954492880415f
/avatars/f3dd763d8e092f68cbc33469f148b337.svg
Kazuhito Koishida
user
koishida
66e3ac6869d1083dd958c120
Arthur Bucker
admin_assigned
2024-09-13T07:53:16.472Z
654a9df87f0fd3f2548eba4c
/avatars/324fbf5daa631331413e7851237f2ebd.svg
Arthur Fender Coelho Bucker
user
ArthurBucker
66e3ac6869d1083dd958c121
Lawrence Jang
claimed_verified
2024-09-13T07:20:50.603Z
664aebe829eadb3ab4e4ca3f
/avatars/548d5656e082c8959ac78b883f0805af.svg
Lawrence Jang
user
ljang0
66e3ac6869d1083dd958c122
Zack Hui
claimed_verified
2025-05-09T07:23:50.509Z
6668c40d6da8706f8d531b5f
/avatars/e0f05182e5e2d8fd101c9548276a1181.svg
zack
user
zackhatecoding
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Large language models (LLMs) show remarkable potential to act as computer
agents, enhancing human productivity and software accessibility in multi-modal
tasks that require planning and reasoning. However, measuring agent performance
in realistic environments remains a challenge since: (i) most benchmarks are
limited to specific modalities or domains (e.g. text-only, web navigation, Q&A,
coding) and (ii) full benchmark evaluations are slow (on order of magnitude of
days) given the multi-step sequential nature of tasks. To address these
challenges, we introduce the Windows Agent Arena: a reproducible, general
environment focusing exclusively on the Windows operating system (OS) where
agents can operate freely within a real Windows OS and use the same wide range
of applications, tools, and web browsers available to human users when solving
tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse
Windows tasks across representative domains that require agent abilities in
planning, screen understanding, and tool usage. Our benchmark is scalable and
can be seamlessly parallelized in Azure for a full benchmark evaluation in as
little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we
also introduce a new multi-modal agent, Navi. Our agent achieves a success rate
of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted
human. Navi also demonstrates strong performance on another popular web-based
benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis
of Navi's performance, and provide insights into the opportunities for future
research in agent development and data generation using Windows Agent Arena.
  Webpage: https://microsoft.github.io/WindowsAgentArena
  Code: https://github.com/microsoft/WindowsAgentArena
Windows Agent Arena provides a general, reproducible environment for evaluating multi-modal agent performance on Windows OS tasks, with scalability and parallelizability features.
LLMs computer agents human productivity software accessibility multi-modal tasks agent performance realistic environments benchmarks modalities domains planning reasoning Windows operating system OSWorld framework screen understanding tool usage scalable parallelizable Azure multi-modal agent Navi success rate Mind2Web quantitative analysis qualitative analysis future research