ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Building Graphical User Interface (GUI) assistants holds significant promise
for enhancing human workflow productivity. While most agents are
language-based, relying on closed-source API with text-rich meta-information
(e.g., HTML or accessibility tree), they show limitations in perceiving UI
visuals as humans do, highlighting the need for GUI visual agents. In this
work, we develop a vision-language-action model in digital world, namely
ShowUI, which features the following innovations: (i) UI-Guided Visual Token
Selection to reduce computational costs by formulating screenshots as an UI
connected graph, adaptively identifying their redundant relationship and serve
as the criteria for token selection during self-attention blocks; (ii)
Interleaved Vision-Language-Action Streaming that flexibly unifies diverse
needs within GUI tasks, enabling effective management of visual-action history
in navigation or pairing multi-turn query-action sequences per screenshot to
enhance training efficiency; (iii) Small-scale High-quality GUI
Instruction-following Datasets by careful data curation and employing a
resampling strategy to address significant data type imbalances. With above
components, ShowUI, a lightweight 2B model using 256K data, achieves a strong
75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection
further reduces 33% of redundant visual tokens during training and speeds up
the performance by 1.4x. Navigation experiments across web Mind2Web, mobile
AITW, and online MiniWob environments further underscore the effectiveness and
potential of our model in advancing GUI visual agents. The models are available
at https://github.com/showlab/ShowUI.
67468ee8e36c38e099c5c0d1
Kevin Qinghong Lin
claimed_verified
2024-11-27T22:20:16.879Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
67468ee8e36c38e099c5c0d2
Linjie Li
67468ee8e36c38e099c5c0d3
Difei Gao
67468ee8e36c38e099c5c0d4
Zhengyuan Yang
67468ee8e36c38e099c5c0d5
Shiwei Wu
67468ee8e36c38e099c5c0d6
Zechen Bai
claimed_verified
2025-03-19T09:48:32.635Z
64b7833aa5018e3c7c9b50d8
/avatars/782415605ed786b73f484fcc86a6384f.svg
Zechen Bai
user
ZechenBai
67468ee8e36c38e099c5c0d7
Weixian Lei
67468ee8e36c38e099c5c0d8
Lijuan Wang
67468ee8e36c38e099c5c0d9
Mike Zheng Shou
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Building Graphical User Interface (GUI) assistants holds significant promise
for enhancing human workflow productivity. While most agents are
language-based, relying on closed-source API with text-rich meta-information
(e.g., HTML or accessibility tree), they show limitations in perceiving UI
visuals as humans do, highlighting the need for GUI visual agents. In this
work, we develop a vision-language-action model in digital world, namely
ShowUI, which features the following innovations: (i) UI-Guided Visual Token
Selection to reduce computational costs by formulating screenshots as an UI
connected graph, adaptively identifying their redundant relationship and serve
as the criteria for token selection during self-attention blocks; (ii)
Interleaved Vision-Language-Action Streaming that flexibly unifies diverse
needs within GUI tasks, enabling effective management of visual-action history
in navigation or pairing multi-turn query-action sequences per screenshot to
enhance training efficiency; (iii) Small-scale High-quality GUI
Instruction-following Datasets by careful data curation and employing a
resampling strategy to address significant data type imbalances. With above
components, ShowUI, a lightweight 2B model using 256K data, achieves a strong
75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection
further reduces 33% of redundant visual tokens during training and speeds up
the performance by 1.4x. Navigation experiments across web Mind2Web, mobile
AITW, and online MiniWob environments further underscore the effectiveness and
potential of our model in advancing GUI visual agents. The models are available
at https://github.com/showlab/ShowUI.
ShowUI is a vision-language-action model that enhances GUI assistants by using UI-guided token selection and interleaved vision-language-action streaming, achieving high accuracy and efficiency in zero-shot screenshot grounding across different environments.
vision-language-action model UI-Guided Visual Token Selection Interleaved Vision-Language-Action Streaming screenshot grounding zero-shot navigation experiments Mind2Web AITW MiniWob resampling strategy token selection self-attention blocks