UFO2: The Desktop AgentOS
Recent Computer-Using Agents (CUAs), powered by multimodal large language
models (LLMs), offer a promising direction for automating complex desktop
workflows through natural language. However, most existing CUAs remain
conceptual prototypes, hindered by shallow OS integration, fragile
screenshot-based interaction, and disruptive execution.
  We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
into practical, system-level automation. UFO2 features a centralized HostAgent
for task decomposition and coordination, alongside a collection of
application-specialized AppAgent equipped with native APIs, domain-specific
knowledge, and a unified GUI--API action layer. This architecture enables
robust task execution while preserving modularity and extensibility. A hybrid
control detection pipeline fuses Windows UI Automation (UIA) with vision-based
parsing to support diverse interface styles. Runtime efficiency is further
enhanced through speculative multi-action planning, reducing per-step LLM
overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
within an isolated virtual desktop, allowing agents and users to operate
concurrently without interference.
  We evaluate UFO2 across over 20 real-world Windows applications,
demonstrating substantial improvements in robustness and execution accuracy
over prior CUAs. Our results show that deep OS integration unlocks a scalable
path toward reliable, user-aligned desktop automation.
6806fab1b89f3c89c81afbe0
Chaoyun Zhang
claimed_verified
2025-04-22T09:51:25.217Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
6806fab1b89f3c89c81afbe1
He Huang
6806fab1b89f3c89c81afbe2
Chiming Ni
6806fab1b89f3c89c81afbe3
Jian Mu
6806fab1b89f3c89c81afbe4
Si Qin
6806fab1b89f3c89c81afbe5
Shilin He
admin_assigned
2025-04-22T10:57:38.634Z
62c6df026a092eda1f1ab6e5
/avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
Shilin He
user
shilhe
6806fab1b89f3c89c81afbe6
Lu Wang
6806fab1b89f3c89c81afbe7
Fangkai Yang
6806fab1b89f3c89c81afbe8
Pu Zhao
6806fab1b89f3c89c81afbe9
Chao Du
admin_assigned
2025-04-22T10:57:25.179Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
6806fab1b89f3c89c81afbea
Liqun Li
admin_assigned
2025-04-22T10:57:10.526Z
666933c97bf97e24f7b5266e
/avatars/283961b37d463a386b08ad33dacca0f4.svg
Liqun Li
user
liqul
6806fab1b89f3c89c81afbeb
Yu Kang
6806fab1b89f3c89c81afbec
Zhao Jiang
6806fab1b89f3c89c81afbed
Suzhen Zheng
6806fab1b89f3c89c81afbee
Rujia Wang
admin_assigned
2025-04-22T10:56:40.486Z
6430e3d1cd31d174a9f9f097
/avatars/132d28d4336ed93b676123f3fb15bd61.svg
rujiawang
user
rujiawang
6806fab1b89f3c89c81afbef
Jiaxu Qian
6806fab1b89f3c89c81afbf0
Minghua Ma
admin_assigned
2025-04-22T10:56:25.955Z
653b52098abd634b83f4fd38
/avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
MingHua Ma
user
Gezelligheid520
6806fab1b89f3c89c81afbf1
Jian-Guang Lou
admin_assigned
2025-04-22T10:56:18.953Z
6554338ba4de813482e91ad5
/avatars/bee879fa4a818fd6c070ee001769ce05.svg
Jian-Guang Lou
user
substill
6806fab1b89f3c89c81afbf2
Qingwei Lin
6806fab1b89f3c89c81afbf3
Saravan Rajmohan
6806fab1b89f3c89c81afbf4
Dongmei Zhang
admin_assigned
2025-04-22T10:56:01.402Z
66473d2c7abe6ad66e81a3dd
/avatars/82f40244806c06ffeaa1c4265e9725ea.svg
ZHANGDONGMEI
user
ZDM6426
UFO2: The Desktop AgentOS
Recent Computer-Using Agents (CUAs), powered by multimodal large language
models (LLMs), offer a promising direction for automating complex desktop
workflows through natural language. However, most existing CUAs remain
conceptual prototypes, hindered by shallow OS integration, fragile
screenshot-based interaction, and disruptive execution.
  We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
into practical, system-level automation. UFO2 features a centralized HostAgent
for task decomposition and coordination, alongside a collection of
application-specialized AppAgent equipped with native APIs, domain-specific
knowledge, and a unified GUI--API action layer. This architecture enables
robust task execution while preserving modularity and extensibility. A hybrid
control detection pipeline fuses Windows UI Automation (UIA) with vision-based
parsing to support diverse interface styles. Runtime efficiency is further
enhanced through speculative multi-action planning, reducing per-step LLM
overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
within an isolated virtual desktop, allowing agents and users to operate
concurrently without interference.
  We evaluate UFO2 across over 20 real-world Windows applications,
demonstrating substantial improvements in robustness and execution accuracy
over prior CUAs. Our results show that deep OS integration unlocks a scalable
path toward reliable, user-aligned desktop automation.
multimodal large language models task decomposition coordination application-specialized AppAgent native APIs domain-specific knowledge unified GUI--API action layer hybrid control detection pipeline Windows UI Automation (UIA) vision-based parsing speculative multi-action planning Picture-in-Picture (PiP) interface runtime efficiency