Towards Trustworthy GUI Agents: A Survey
GUI agents, powered by large foundation models, can interact with digital
interfaces, enabling various applications in web automation, mobile navigation,
and software testing. However, their increasing autonomy has raised critical
concerns about their security, privacy, and safety. This survey examines the
trustworthiness of GUI agents in five critical dimensions: security
vulnerabilities, reliability in dynamic environments, transparency and
explainability, ethical considerations, and evaluation methodologies. We also
identify major challenges such as vulnerability to adversarial attacks,
cascading failure modes in sequential decision-making, and a lack of realistic
evaluation benchmarks. These issues not only hinder real-world deployment but
also call for comprehensive mitigation strategies beyond task success. As GUI
agents become more widespread, establishing robust safety standards and
responsible development practices is essential. This survey provides a
foundation for advancing trustworthy GUI agents through systematic
understanding and future research.
67ecf47dd0f4d6684e0fb7e4
Yucheng Shi
admin_assigned
2025-04-02T12:28:32.523Z
64beb6b6140491ca9f803ebf
/avatars/0daa2e813a13668b8b708cd8c12763d9.svg
Yucheng SHi
user
YuchengShi
67ecf47dd0f4d6684e0fb7e5
Wenhao Yu
67ecf47dd0f4d6684e0fb7e6
Wenlin Yao
admin_assigned
2025-04-02T12:28:50.560Z
634f18e4aae4bde2c8e2adca
/avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
Wenlin Yao
user
wenlinyao
67ecf47dd0f4d6684e0fb7e7
Wenhu Chen
admin_assigned
2025-04-02T12:28:56.387Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
67ecf47dd0f4d6684e0fb7e8
Ninghao Liu
admin_assigned
2025-04-02T12:29:04.216Z
67baa6c519e9dba50ece56b1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/F3ZmRr7nCzfSVbpS4SY1E.png
Ninghao Liu
user
NeoLiu43
Towards Trustworthy GUI Agents: A Survey
GUI agents, powered by large foundation models, can interact with digital
interfaces, enabling various applications in web automation, mobile navigation,
and software testing. However, their increasing autonomy has raised critical
concerns about their security, privacy, and safety. This survey examines the
trustworthiness of GUI agents in five critical dimensions: security
vulnerabilities, reliability in dynamic environments, transparency and
explainability, ethical considerations, and evaluation methodologies. We also
identify major challenges such as vulnerability to adversarial attacks,
cascading failure modes in sequential decision-making, and a lack of realistic
evaluation benchmarks. These issues not only hinder real-world deployment but
also call for comprehensive mitigation strategies beyond task success. As GUI
agents become more widespread, establishing robust safety standards and
responsible development practices is essential. This survey provides a
foundation for advancing trustworthy GUI agents through systematic
understanding and future research.
GUI agents powered by large foundation models interact with digital interfaces but face challenges in security, reliability, transparency, ethics, and evaluation.
GUI agents large foundation models security vulnerabilities reliability dynamic environments transparency explainability ethical considerations evaluation methodologies adversarial attacks cascading failure modes evaluation benchmarks