UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Multimodal agents for computer use rely exclusively on primitive actions
(click, type, scroll) that require accurate visual grounding and lengthy
execution chains, leading to cascading failures and performance bottlenecks.
While other agents leverage rich programmatic interfaces (APIs, MCP servers,
tools), computer-use agents (CUAs) remain isolated from these capabilities. We
present UltraCUA, a foundation model that bridges this gap through hybrid
action -- seamlessly integrating GUI primitives with high-level programmatic
tool calls. To achieve this, our approach comprises four key components: (1) an
automated pipeline that scales programmatic tools from software documentation,
open-source repositories, and code generation; (2) a synthetic data engine
producing over 17,000 verifiable tasks spanning real-world computer-use
scenarios; (3) a large-scale high-quality hybrid action trajectory collection
with both low-level GUI actions and high-level programmatic tool calls; and (4)
a two-stage training pipeline combining supervised fine-tuning with online
reinforcement learning, enabling strategic alternation between low-level and
high-level actions. Experiments with our 7B and 32B models demonstrate
substantial improvements over state-of-the-art agents. On OSWorld, UltraCUA
models achieve an average 22% relative improvement over base models, while
being 11% faster in terms of steps. Out-of-domain evaluation on
WindowsAgentArena shows our model reaches 21.7% success rate, outperforming
baselines trained on Windows data. The hybrid action mechanism proves critical,
reducing error propagation while maintaining execution efficiency.
68f6ff6224c4489363111976
Yuhao Yang
68f6ff6224c4489363111977
Zhen Yang
68f6ff6224c4489363111978
Zi-Yi Dou
68f6ff6224c4489363111979
Anh Nguyen
claimed_verified
2025-10-22T15:45:49.114Z
61ee4dc39539bc0a3bbc8a68
/avatars/723f89e9c7afecdbd2854544bd7eb083.svg
Anh Tuan Nguyen
user
tuananh7198
68f6ff6224c448936311197a
Keen You
68f6ff6224c448936311197b
Omar Attia
68f6ff6224c448936311197c
Andrew Szot
68f6ff6224c448936311197d
Michael Feng
68f6ff6224c448936311197e
Ram Ramrakhya
68f6ff6224c448936311197f
Alexander Toshev
68f6ff6224c4489363111980
Chao Huang
68f6ff6224c4489363111981
Yinfei Yang
68f6ff6224c4489363111982
Zhe Gan
UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Multimodal agents for computer use rely exclusively on primitive actions
(click, type, scroll) that require accurate visual grounding and lengthy
execution chains, leading to cascading failures and performance bottlenecks.
While other agents leverage rich programmatic interfaces (APIs, MCP servers,
tools), computer-use agents (CUAs) remain isolated from these capabilities. We
present UltraCUA, a foundation model that bridges this gap through hybrid
action -- seamlessly integrating GUI primitives with high-level programmatic
tool calls. To achieve this, our approach comprises four key components: (1) an
automated pipeline that scales programmatic tools from software documentation,
open-source repositories, and code generation; (2) a synthetic data engine
producing over 17,000 verifiable tasks spanning real-world computer-use
scenarios; (3) a large-scale high-quality hybrid action trajectory collection
with both low-level GUI actions and high-level programmatic tool calls; and (4)
a two-stage training pipeline combining supervised fine-tuning with online
reinforcement learning, enabling strategic alternation between low-level and
high-level actions. Experiments with our 7B and 32B models demonstrate
substantial improvements over state-of-the-art agents. On OSWorld, UltraCUA
models achieve an average 22% relative improvement over base models, while
being 11% faster in terms of steps. Out-of-domain evaluation on
WindowsAgentArena shows our model reaches 21.7% success rate, outperforming
baselines trained on Windows data. The hybrid action mechanism proves critical,
reducing error propagation while maintaining execution efficiency.
UltraCUA integrates GUI actions with programmatic tools to improve computer-use agent performance and efficiency.
GUI primitives programmatic interfaces APIs MCP servers tools foundation model hybrid action automated pipeline synthetic data engine high-quality hybrid action trajectory supervised fine-tuning online reinforcement learning OSWorld WindowsAgentArena