OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents
In this paper, we introduce OSUniverse: a benchmark of complex, multimodal
desktop-oriented tasks for advanced GUI-navigation AI agents that focuses on
ease of use, extensibility, comprehensive coverage of test cases, and automated
validation. We divide the tasks in increasing levels of complexity, from basic
precision clicking to multistep, multiapplication tests requiring dexterity,
precision, and clear thinking from the agent. In version one of the benchmark,
presented here, we have calibrated the complexity of the benchmark test cases
to ensure that the SOTA (State of the Art) agents (at the time of publication)
do not achieve results higher than 50%, while the average white collar worker
can perform all these tasks with perfect accuracy. The benchmark can be scored
manually, but we also introduce an automated validation mechanism that has an
average error rate less than 2%. Therefore, this benchmark presents solid
ground for fully automated measuring of progress, capabilities and the
effectiveness of GUI-navigation AI agents over the short and medium-term
horizon. The source code of the benchmark is available at
https://github.com/agentsea/osuniverse.
681b518bf497fd5e45b55eeb
Mariya Davydova
admin_assigned
2025-05-14T11:27:43.146Z
62d16ec939e0332b574d0d95
/avatars/52a69820fffb347b3ea495b121bb3ad9.svg
Mariya Davydova
user
mariyadavydova
681b518bf497fd5e45b55eec
Daniel Jeffries
681b518bf497fd5e45b55eed
Patrick Barker
681b518bf497fd5e45b55eee
Arturo Márquez Flores
681b518bf497fd5e45b55eef
Sinéad Ryan
OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents
In this paper, we introduce OSUniverse: a benchmark of complex, multimodal
desktop-oriented tasks for advanced GUI-navigation AI agents that focuses on
ease of use, extensibility, comprehensive coverage of test cases, and automated
validation. We divide the tasks in increasing levels of complexity, from basic
precision clicking to multistep, multiapplication tests requiring dexterity,
precision, and clear thinking from the agent. In version one of the benchmark,
presented here, we have calibrated the complexity of the benchmark test cases
to ensure that the SOTA (State of the Art) agents (at the time of publication)
do not achieve results higher than 50%, while the average white collar worker
can perform all these tasks with perfect accuracy. The benchmark can be scored
manually, but we also introduce an automated validation mechanism that has an
average error rate less than 2%. Therefore, this benchmark presents solid
ground for fully automated measuring of progress, capabilities and the
effectiveness of GUI-navigation AI agents over the short and medium-term
horizon. The source code of the benchmark is available at
https://github.com/agentsea/osuniverse.
OSUniverse benchmarks multimodal desktop tasks for GUI-navigation AI, offering automated validation and scoring mechanisms.
OSUniverse GUI-navigation AI benchmark multimodal tasks desktop-oriented automated validation accuracy complexity calibration