Navigating the Digital World as Humans Do: Universal Visual Grounding
  for GUI Agents
Multimodal large language models (MLLMs) are transforming the capabilities of
graphical user interface (GUI) agents, facilitating their transition from
controlled simulations to complex, real-world applications across various
platforms. However, the effectiveness of these agents hinges on the robustness
of their grounding capability. Current GUI agents predominantly utilize
text-based representations such as HTML or accessibility trees, which, despite
their utility, often introduce noise, incompleteness, and increased
computational overhead. In this paper, we advocate a human-like embodiment for
GUI agents that perceive the environment entirely visually and directly take
pixel-level operations on the GUI. The key is visual grounding models that can
accurately map diverse referring expressions of GUI elements to their
coordinates on the GUI across different platforms. We show that a simple
recipe, which includes web-based synthetic data and slight adaptation of the
LLaVA architecture, is surprisingly effective for training such visual
grounding models. We collect the largest dataset for GUI visual grounding so
far, containing 10M GUI elements and their referring expressions over 1.3M
screenshots, and use it to train UGround, a strong universal visual grounding
model for GUI agents. Empirical results on six benchmarks spanning three
categories (grounding, offline agent, and online agent) show that 1) UGround
substantially outperforms existing visual grounding models for GUI agents, by
up to 20% absolute, and 2) agents with UGround outperform state-of-the-art
agents, despite the fact that existing agents use additional text-based input
while ours only uses visual perception. These results provide strong support
for the feasibility and promises of GUI agents that navigate the digital world
as humans do.
6704ac58fdaeb32c253487c2
Boyu Gou
claimed_verified
2024-10-08T15:17:32.360Z
6500870f1e14749e84f8f887
https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
Boyu Gou
user
BoyuNLP
6704ac58fdaeb32c253487c3
Ruohan Wang
extracted_pending
2024-10-08T03:51:54.906Z
649d00587016f9f2fcbc0fd9
/avatars/4f2e3e239bb19c1597463f2d5010fb7d.svg
Yanan Xie
user
orby-yanan
6704ac58fdaeb32c253487c4
Boyuan Zheng
claimed_verified
2024-10-10T08:02:04.746Z
631a95cfa66151e36e54a905
https://cdn-avatars.huggingface.co/v1/production/uploads/631a95cfa66151e36e54a905/jM_4vy9_Kd8zomFVYtsTo.png
Boyuan Zheng
user
boyuanzheng010
6704ac58fdaeb32c253487c5
Yanan Xie
6704ac58fdaeb32c253487c6
Cheng Chang
6704ac58fdaeb32c253487c7
Yiheng Shu
claimed_verified
2024-10-08T15:17:28.570Z
60a4ebfbaa9320dbbe69e37c
https://cdn-avatars.huggingface.co/v1/production/uploads/60a4ebfbaa9320dbbe69e37c/QLaEohXCWaUy8YX3wKQ_w.jpeg
Yiheng Shu
user
yhshu
6704ac58fdaeb32c253487c8
Huan Sun
6704ac58fdaeb32c253487c9
Yu Su
claimed_verified
2024-10-08T15:17:30.464Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
Navigating the Digital World as Humans Do: Universal Visual Grounding
  for GUI Agents
Multimodal large language models (MLLMs) are transforming the capabilities of
graphical user interface (GUI) agents, facilitating their transition from
controlled simulations to complex, real-world applications across various
platforms. However, the effectiveness of these agents hinges on the robustness
of their grounding capability. Current GUI agents predominantly utilize
text-based representations such as HTML or accessibility trees, which, despite
their utility, often introduce noise, incompleteness, and increased
computational overhead. In this paper, we advocate a human-like embodiment for
GUI agents that perceive the environment entirely visually and directly take
pixel-level operations on the GUI. The key is visual grounding models that can
accurately map diverse referring expressions of GUI elements to their
coordinates on the GUI across different platforms. We show that a simple
recipe, which includes web-based synthetic data and slight adaptation of the
LLaVA architecture, is surprisingly effective for training such visual
grounding models. We collect the largest dataset for GUI visual grounding so
far, containing 10M GUI elements and their referring expressions over 1.3M
screenshots, and use it to train UGround, a strong universal visual grounding
model for GUI agents. Empirical results on six benchmarks spanning three
categories (grounding, offline agent, and online agent) show that 1) UGround
substantially outperforms existing visual grounding models for GUI agents, by
up to 20% absolute, and 2) agents with UGround outperform state-of-the-art
agents, despite the fact that existing agents use additional text-based input
while ours only uses visual perception. These results provide strong support
for the feasibility and promises of GUI agents that navigate the digital world
as humans do.
Visual grounding models for GUI agents using pixel-level operations outperform text-based methods, demonstrating the feasibility of human-like visual perception in complex applications.
multimodal large language models GUI agents visual grounding models pixel-level operations referring expressions synthetic data LLaVA architecture universal visual grounding model GUI visual grounding dataset offline agent online agent