LLM-Powered GUI Agents in Phone Automation: Surveying Progress and
  Prospects
With the rapid rise of large language models (LLMs), phone automation has
undergone transformative changes. This paper systematically reviews LLM-driven
phone GUI agents, highlighting their evolution from script-based automation to
intelligent, adaptive systems. We first contextualize key challenges, (i)
limited generality, (ii) high maintenance overhead, and (iii) weak intent
comprehension, and show how LLMs address these issues through advanced language
understanding, multimodal perception, and robust decision-making. We then
propose a taxonomy covering fundamental agent frameworks (single-agent,
multi-agent, plan-then-act), modeling approaches (prompt engineering,
training-based), and essential datasets and benchmarks. Furthermore, we detail
task-specific architectures, supervised fine-tuning, and reinforcement learning
strategies that bridge user intent and GUI operations. Finally, we discuss open
challenges such as dataset diversity, on-device deployment efficiency,
user-centric adaptation, and security concerns, offering forward-looking
insights into this rapidly evolving field. By providing a structured overview
and identifying pressing research gaps, this paper serves as a definitive
reference for researchers and practitioners seeking to harness LLMs in
designing scalable, user-friendly phone GUI agents.
6810317e007d579cbf5200ba
Guangyi Liu
admin_assigned
2025-04-29T12:15:01.850Z
6108ae87823007eaf0c7bd1e
https://cdn-avatars.huggingface.co/v1/production/uploads/6108ae87823007eaf0c7bd1e/dKjdx9I5waJs6oUQ0_mmT.png
Guangyi Liu
user
guangyil
6810317e007d579cbf5200bb
Pengxiang Zhao
claimed_verified
2025-04-29T07:59:51.686Z
65a088f4300957620ba45c70
/avatars/56ed45e10d3455531979f30881b2d3f9.svg
pengxiang zhao
user
Pengxiangzhao
6810317e007d579cbf5200bc
Liang Liu
claimed_verified
2025-04-29T08:01:16.707Z
667b91162bfe908436900faa
/avatars/daeaf058ec1df5307996895a5cbba052.svg
Liang Liu
user
melpancake
6810317e007d579cbf5200bd
Yaxuan Guo
admin_assigned
2025-04-29T12:15:15.455Z
65c237220c57a7141888363e
/avatars/ce43c52f47d524c5b747523058946325.svg
guoyaxuan
user
guoyaxuan0106
6810317e007d579cbf5200be
Han Xiao
claimed_verified
2025-05-28T09:01:47.021Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
6810317e007d579cbf5200bf
Weifeng Lin
admin_assigned
2025-04-29T12:15:29.782Z
66026c9068d519ed32519e9c
/avatars/8fa051312c713772e5b8ba65989ff7f5.svg
Weifeng Lin
user
Afeng-x
6810317e007d579cbf5200c0
Yuxiang Chai
claimed_verified
2025-04-29T08:01:19.091Z
6458ce236fa580137af5aa95
/avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
Yuxiang Chai
user
Yuxiang007
6810317e007d579cbf5200c1
Yue Han
6810317e007d579cbf5200c2
Shuai Ren
6810317e007d579cbf5200c3
Hao Wang
6810317e007d579cbf5200c4
Xiaoyu Liang
6810317e007d579cbf5200c5
Wenhao Wang
6810317e007d579cbf5200c6
Tianze Wu
6810317e007d579cbf5200c7
Linghao Li
6810317e007d579cbf5200c8
Hao Wang
6810317e007d579cbf5200c9
Guanjing Xiong
6810317e007d579cbf5200ca
Yong Liu
6810317e007d579cbf5200cb
Hongsheng Li
admin_assigned
2025-04-29T12:17:24.303Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
LLM-Powered GUI Agents in Phone Automation: Surveying Progress and
  Prospects
With the rapid rise of large language models (LLMs), phone automation has
undergone transformative changes. This paper systematically reviews LLM-driven
phone GUI agents, highlighting their evolution from script-based automation to
intelligent, adaptive systems. We first contextualize key challenges, (i)
limited generality, (ii) high maintenance overhead, and (iii) weak intent
comprehension, and show how LLMs address these issues through advanced language
understanding, multimodal perception, and robust decision-making. We then
propose a taxonomy covering fundamental agent frameworks (single-agent,
multi-agent, plan-then-act), modeling approaches (prompt engineering,
training-based), and essential datasets and benchmarks. Furthermore, we detail
task-specific architectures, supervised fine-tuning, and reinforcement learning
strategies that bridge user intent and GUI operations. Finally, we discuss open
challenges such as dataset diversity, on-device deployment efficiency,
user-centric adaptation, and security concerns, offering forward-looking
insights into this rapidly evolving field. By providing a structured overview
and identifying pressing research gaps, this paper serves as a definitive
reference for researchers and practitioners seeking to harness LLMs in
designing scalable, user-friendly phone GUI agents.
LLM-driven phone GUI agents evolve from script-based automation to intelligent systems, using advanced language understanding, multimodal perception, and robust decision-making to address generality, maintenance overhead, and intent comprehension challenges.
LLMs phone GUI agents script-based automation intelligent systems advanced language understanding multimodal perception robust decision-making single-agent multi-agent plan-then-act prompt engineering training-based supervised fine-tuning reinforcement learning user intent GUI operations dataset diversity on-device deployment efficiency user-centric adaptation security concerns