Agent S2: A Compositional Generalist-Specialist Framework for Computer
  Use Agents
Computer use agents automate digital tasks by directly interacting with
graphical user interfaces (GUIs) on computers and mobile devices, offering
significant potential to enhance human productivity by completing an open-ended
space of user queries. However, current agents face significant challenges:
imprecise grounding of GUI elements, difficulties with long-horizon task
planning, and performance bottlenecks from relying on single generalist models
for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
compositional framework that delegates cognitive responsibilities across
various generalist and specialist models. We propose a novel
Mixture-of-Grounding technique to achieve precise GUI localization and
introduce Proactive Hierarchical Planning, dynamically refining action plans at
multiple temporal scales in response to evolving observations. Evaluations
demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
on three prominent computer use benchmarks. Specifically, Agent S2 achieves
18.9% and 32.7% relative improvements over leading baseline agents such as
Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
Moreover, Agent S2 generalizes effectively to other operating systems and
applications, surpassing previous best methods by 52.8% on WindowsAgentArena
and by 16.52% on AndroidWorld relatively. Code available at
https://github.com/simular-ai/Agent-S.
67ec9340913c072638c16bbf
Saaket Agashe
admin_assigned
2025-04-02T09:49:55.477Z
627837664f2afdc41bbd622a
/avatars/b7bfc4fb77830bba71839c04a4aeea64.svg
Saaket Agashe
user
saa1605
67ec9340913c072638c16bc0
Kyle Wong
claimed_verified
2025-04-15T08:01:17.795Z
6526e6a4329ee9e7cb3045e5
/avatars/ee9c10d6bd6b3fc96313cb538c94f678.svg
Kyle Wong
user
ewveggies
67ec9340913c072638c16bc1
Vincent Tu
67ec9340913c072638c16bc2
Jiachen Yang
admin_assigned
2025-04-02T09:46:55.173Z
65edff6233c279253952e0bd
/avatars/3130f6f9873ae0a943631e56f6d8d341.svg
Jiachen Yang
user
jc-y42
67ec9340913c072638c16bc3
Ang Li
67ec9340913c072638c16bc4
Xin Eric Wang
admin_assigned
2025-04-02T09:46:30.920Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
Agent S2: A Compositional Generalist-Specialist Framework for Computer
  Use Agents
Computer use agents automate digital tasks by directly interacting with
graphical user interfaces (GUIs) on computers and mobile devices, offering
significant potential to enhance human productivity by completing an open-ended
space of user queries. However, current agents face significant challenges:
imprecise grounding of GUI elements, difficulties with long-horizon task
planning, and performance bottlenecks from relying on single generalist models
for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
compositional framework that delegates cognitive responsibilities across
various generalist and specialist models. We propose a novel
Mixture-of-Grounding technique to achieve precise GUI localization and
introduce Proactive Hierarchical Planning, dynamically refining action plans at
multiple temporal scales in response to evolving observations. Evaluations
demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
on three prominent computer use benchmarks. Specifically, Agent S2 achieves
18.9% and 32.7% relative improvements over leading baseline agents such as
Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
Moreover, Agent S2 generalizes effectively to other operating systems and
applications, surpassing previous best methods by 52.8% on WindowsAgentArena
and by 16.52% on AndroidWorld relatively. Code available at
https://github.com/simular-ai/Agent-S.
Agent S2, a compositional framework using Mixture-of-Grounding and Proactive Hierarchical Planning, achieves state-of-the-art performance in computer use automation across various benchmarks and operating systems.
Mixture-of-Grounding Proactive Hierarchical Planning compositional framework