OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Existing efforts in building GUI agents heavily rely on the availability of
robust commercial Vision-Language Models (VLMs) such as GPT-4o and
GeminiProVision. Practitioners are often reluctant to use open-source VLMs due
to their significant performance lag compared to their closed-source
counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD)
scenarios. To facilitate future research in this area, we developed OS-Atlas -
a foundational GUI action model that excels at GUI grounding and OOD agentic
tasks through innovations in both data and modeling. We have invested
significant engineering effort in developing an open-source toolkit for
synthesizing GUI grounding data across multiple platforms, including Windows,
Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing
the largest open-source cross-platform GUI grounding corpus to date, which
contains over 13 million GUI elements. This dataset, combined with innovations
in model training, provides a solid foundation for OS-Atlas to understand GUI
screenshots and generalize to unseen interfaces. Through extensive evaluation
across six benchmarks spanning three different platforms (mobile, desktop, and
web), OS-Atlas demonstrates significant performance improvements over previous
state-of-the-art models. Our evaluation also uncovers valuable insights into
continuously improving and scaling the agentic capabilities of open-source
VLMs.
67286aaf6c2c5bca2617d7b4
Zhiyong Wu
admin_assigned
2024-11-04T14:18:19.133Z
6280e830e99dccaac4bbfde5
/avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
Zhiyong Wu
user
zy001
67286aaf6c2c5bca2617d7b5
Zhenyu Wu
67286aaf6c2c5bca2617d7b6
Fangzhi Xu
admin_assigned
2024-11-04T14:18:52.631Z
64e6cf78ecce34cb442dc889
https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
Fangzhi Xu
user
xufangzhi
67286aaf6c2c5bca2617d7b7
Yian Wang
67286aaf6c2c5bca2617d7b8
Qiushi Sun
claimed_verified
2024-11-04T11:37:21.047Z
6064a0eeb1703ddba0d458b9
https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
Qiushi
user
QiushiSun
67286aaf6c2c5bca2617d7b9
Chengyou Jia
admin_assigned
2024-11-04T14:19:11.993Z
6602548a68d519ed324b47c5
/avatars/5ab411f87440cc2a98c7a1c6a3ed5548.svg
ChengyouJia
user
ChengyouJia
67286aaf6c2c5bca2617d7ba
Kanzhi Cheng
admin_assigned
2024-11-04T14:19:17.294Z
63340dbbd92c5842ae71d1e9
/avatars/3a3182996bd41b526dcbfa8687d91963.svg
Kanzhi Cheng
user
cckevinn
67286aaf6c2c5bca2617d7bb
Zichen Ding
claimed_verified
2024-11-04T09:35:44.330Z
642b9861bb77f8456634b048
https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/VrNmmcdgX7FufQmdP5YaG.jpeg
Zichen Ding
user
heroding77
67286aaf6c2c5bca2617d7bc
Liheng Chen
admin_assigned
2024-11-04T14:19:29.363Z
6561824484a9fbe322b9abc3
https://cdn-avatars.huggingface.co/v1/production/uploads/6561824484a9fbe322b9abc3/omS3D6PzGBD7kc4S7bOIO.png
LIHENG CHEN
user
Lemaqwq
67286aaf6c2c5bca2617d7bd
Paul Pu Liang
67286aaf6c2c5bca2617d7be
Yu Qiao
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Existing efforts in building GUI agents heavily rely on the availability of
robust commercial Vision-Language Models (VLMs) such as GPT-4o and
GeminiProVision. Practitioners are often reluctant to use open-source VLMs due
to their significant performance lag compared to their closed-source
counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD)
scenarios. To facilitate future research in this area, we developed OS-Atlas -
a foundational GUI action model that excels at GUI grounding and OOD agentic
tasks through innovations in both data and modeling. We have invested
significant engineering effort in developing an open-source toolkit for
synthesizing GUI grounding data across multiple platforms, including Windows,
Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing
the largest open-source cross-platform GUI grounding corpus to date, which
contains over 13 million GUI elements. This dataset, combined with innovations
in model training, provides a solid foundation for OS-Atlas to understand GUI
screenshots and generalize to unseen interfaces. Through extensive evaluation
across six benchmarks spanning three different platforms (mobile, desktop, and
web), OS-Atlas demonstrates significant performance improvements over previous
state-of-the-art models. Our evaluation also uncovers valuable insights into
continuously improving and scaling the agentic capabilities of open-source
VLMs.
OS-Atlas improves GUI agent performance through a large open-source GUI grounding dataset and model innovations.
GUI agents Vision-Language Models (VLMs) GUI grounding Out-Of-Distribution (OOD) open-source toolkit GUI screenshots benchmarks