OS Agents: A Survey on MLLM-based Agents for General Computing Devices
  Use
The dream to create AI assistants as capable and versatile as the fictional
J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution
of (multi-modal) large language models ((M)LLMs), this dream is closer to
reality, as (M)LLM-based Agents using computing devices (e.g., computers and
mobile phones) by operating within the environments and interfaces (e.g.,
Graphical User Interface (GUI)) provided by operating systems (OS) to automate
tasks have significantly advanced. This paper presents a comprehensive survey
of these advanced agents, designated as OS Agents. We begin by elucidating the
fundamentals of OS Agents, exploring their key components including the
environment, observation space, and action space, and outlining essential
capabilities such as understanding, planning, and grounding. We then examine
methodologies for constructing OS Agents, focusing on domain-specific
foundation models and agent frameworks. A detailed review of evaluation
protocols and benchmarks highlights how OS Agents are assessed across diverse
tasks. Finally, we discuss current challenges and identify promising directions
for future research, including safety and privacy, personalization and
self-evolution. This survey aims to consolidate the state of OS Agents
research, providing insights to guide both academic inquiry and industrial
development. An open-source GitHub repository is maintained as a dynamic
resource to foster further innovation in this field. We present a 9-page
version of our work, accepted by ACL 2025, to provide a concise overview to the
domain.
6899fdca72289c9658cf4776
Xueyu Hu
6899fdca72289c9658cf4777
Tao Xiong
6899fdca72289c9658cf4778
Biao Yi
6899fdca72289c9658cf4779
Zishu Wei
6899fdca72289c9658cf477a
Ruixuan Xiao
6899fdca72289c9658cf477b
Yurun Chen
6899fdca72289c9658cf477c
Jiasheng Ye
6899fdca72289c9658cf477d
Meiling Tao
6899fdca72289c9658cf477e
Xiangxin Zhou
6899fdca72289c9658cf477f
Ziyu Zhao
6899fdca72289c9658cf4780
Yuhuai Li
6899fdca72289c9658cf4781
Shengze Xu
6899fdca72289c9658cf4782
Shenzhi Wang
6899fdca72289c9658cf4783
Xinchen Xu
6899fdca72289c9658cf4784
Shuofei Qiao
6899fdca72289c9658cf4785
Zhaokai Wang
6899fdca72289c9658cf4786
Kun Kuang
6899fdca72289c9658cf4787
Tieyong Zeng
6899fdca72289c9658cf4788
Liang Wang
6899fdca72289c9658cf4789
Jiwei Li
6899fdca72289c9658cf478a
Yuchen Eleanor Jiang
6899fdca72289c9658cf478b
Wangchunshu Zhou
6899fdca72289c9658cf478c
Guoyin Wang
6899fdca72289c9658cf478d
Keting Yin
6899fdca72289c9658cf478e
Zhou Zhao
6899fdca72289c9658cf478f
Hongxia Yang
6899fdca72289c9658cf4790
Fan Wu
6899fdca72289c9658cf4791
Shengyu Zhang
6899fdca72289c9658cf4792
Fei Wu
OS Agents: A Survey on MLLM-based Agents for General Computing Devices
  Use
The dream to create AI assistants as capable and versatile as the fictional
J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution
of (multi-modal) large language models ((M)LLMs), this dream is closer to
reality, as (M)LLM-based Agents using computing devices (e.g., computers and
mobile phones) by operating within the environments and interfaces (e.g.,
Graphical User Interface (GUI)) provided by operating systems (OS) to automate
tasks have significantly advanced. This paper presents a comprehensive survey
of these advanced agents, designated as OS Agents. We begin by elucidating the
fundamentals of OS Agents, exploring their key components including the
environment, observation space, and action space, and outlining essential
capabilities such as understanding, planning, and grounding. We then examine
methodologies for constructing OS Agents, focusing on domain-specific
foundation models and agent frameworks. A detailed review of evaluation
protocols and benchmarks highlights how OS Agents are assessed across diverse
tasks. Finally, we discuss current challenges and identify promising directions
for future research, including safety and privacy, personalization and
self-evolution. This survey aims to consolidate the state of OS Agents
research, providing insights to guide both academic inquiry and industrial
development. An open-source GitHub repository is maintained as a dynamic
resource to foster further innovation in this field. We present a 9-page
version of our work, accepted by ACL 2025, to provide a concise overview to the
domain.