GUI Agents: A Survey
Graphical User Interface (GUI) agents, powered by Large Foundation Models,
have emerged as a transformative approach to automating human-computer
interaction. These agents autonomously interact with digital systems or
software applications via GUIs, emulating human actions such as clicking,
typing, and navigating visual elements across diverse platforms. Motivated by
the growing interest and fundamental importance of GUI agents, we provide a
comprehensive survey that categorizes their benchmarks, evaluation metrics,
architectures, and training methods. We propose a unified framework that
delineates their perception, reasoning, planning, and acting capabilities.
Furthermore, we identify important open challenges and discuss key future
directions. Finally, this work serves as a basis for practitioners and
researchers to gain an intuitive understanding of current progress, techniques,
benchmarks, and critical open problems that remain to be addressed.
676389c7c998c60976a9a943
Dang Nguyen
extracted_pending
2024-12-19T02:49:44.513Z
64ba4c2565535cf237da429a
/avatars/a8af0e748686e9d3364f1beaa5039ddb.svg
Dang Nguyen
user
dangmn
676389c7c998c60976a9a944
Jian Chen
claimed_verified
2025-07-03T07:43:50.242Z
64593836c16ecb4815e082ce
https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/T9WM0Ly6ADi3a9JUSIsSV.png
Jian Chen
user
puar-playground
676389c7c998c60976a9a945
Yu Wang
676389c7c998c60976a9a946
Gang Wu
676389c7c998c60976a9a947
Namyong Park
676389c7c998c60976a9a948
Zhengmian Hu
676389c7c998c60976a9a949
Hanjia Lyu
676389c7c998c60976a9a94a
Junda Wu
676389c7c998c60976a9a94b
Ryan Aponte
676389c7c998c60976a9a94c
Yu Xia
676389c7c998c60976a9a94d
Xintong Li
676389c7c998c60976a9a94e
Jing Shi
676389c7c998c60976a9a94f
Hongjie Chen
676389c7c998c60976a9a950
Viet Dac Lai
676389c7c998c60976a9a951
Zhouhang Xie
676389c7c998c60976a9a952
Sungchul Kim
676389c7c998c60976a9a953
Ruiyi Zhang
676389c7c998c60976a9a954
Tong Yu
676389c7c998c60976a9a955
Mehrab Tanjim
676389c7c998c60976a9a956
Nesreen K. Ahmed
claimed_verified
2025-01-07T08:42:51.987Z
663e588ccca86c8371a913d9
https://cdn-avatars.huggingface.co/v1/production/uploads/663e588ccca86c8371a913d9/wgVGw5U77cqwtjovhsXyR.jpeg
Nesreen Ahmed
user
nkahmed
676389c7c998c60976a9a957
Puneet Mathur
676389c7c998c60976a9a958
Seunghyun Yoon
676389c7c998c60976a9a959
Lina Yao
676389c7c998c60976a9a95a
Branislav Kveton
676389c7c998c60976a9a95b
Thien Huu Nguyen
676389c7c998c60976a9a95c
Trung Bui
676389c7c998c60976a9a95d
Tianyi Zhou
claimed_verified
2024-12-19T18:01:31.207Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
676389c7c998c60976a9a95e
Ryan A. Rossi
676389c7c998c60976a9a95f
Franck Dernoncourt
admin_assigned
2024-12-19T13:18:13.285Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
GUI Agents: A Survey
Graphical User Interface (GUI) agents, powered by Large Foundation Models,
have emerged as a transformative approach to automating human-computer
interaction. These agents autonomously interact with digital systems or
software applications via GUIs, emulating human actions such as clicking,
typing, and navigating visual elements across diverse platforms. Motivated by
the growing interest and fundamental importance of GUI agents, we provide a
comprehensive survey that categorizes their benchmarks, evaluation metrics,
architectures, and training methods. We propose a unified framework that
delineates their perception, reasoning, planning, and acting capabilities.
Furthermore, we identify important open challenges and discuss key future
directions. Finally, this work serves as a basis for practitioners and
researchers to gain an intuitive understanding of current progress, techniques,
benchmarks, and critical open problems that remain to be addressed.
A survey of graphical user interface agents powered by large foundation models, detailing benchmarks, metrics, architectures, training methods, and future challenges in automating human-computer interaction.
Large Foundation Models GUI agents human-computer interaction autonomous interaction digital systems software applications clicking typing navigating visual elements perception reasoning planning acting unified framework open challenges future directions