OmniParser for Pure Vision Based GUI Agent
The recent success of large vision language models shows great potential in
driving the agent system operating on user interfaces. However, we argue that
the power multimodal models like GPT-4V as a general agent on multiple
operating systems across different applications is largely underestimated due
to the lack of a robust screen parsing technique capable of: 1) reliably
identifying interactable icons within the user interface, and 2) understanding
the semantics of various elements in a screenshot and accurately associate the
intended action with the corresponding region on the screen. To fill these
gaps, we introduce OmniParser, a comprehensive method for parsing user
interface screenshots into structured elements, which significantly enhances
the ability of GPT-4V to generate actions that can be accurately grounded in
the corresponding regions of the interface. We first curated an interactable
icon detection dataset using popular webpages and an icon description dataset.
These datasets were utilized to fine-tune specialized models: a detection model
to parse interactable regions on the screen and a caption model to extract the
functional semantics of the detected elements. OmniParser
significantly improves GPT-4V's performance on ScreenSpot benchmark. And on
Mind2Web and AITW benchmark, OmniParser with screenshot only input
outperforms the GPT-4V baselines requiring additional information outside of
screenshot.
66ac4558169efd8637e3e4a7
Yadong Lu
66ac4558169efd8637e3e4a8
Jianwei Yang
admin_assigned
2024-08-02T08:22:30.221Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
66ac4558169efd8637e3e4a9
Yelong Shen
admin_assigned
2024-08-02T08:22:37.359Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
66ac4558169efd8637e3e4aa
Ahmed Awadallah
admin_assigned
2024-08-02T08:22:51.189Z
6557a41ee74d1065f7d05c51
/avatars/b29e57fc61d1d3012637f84b82c14818.svg
Ahmed Awadallah
user
AhmedAwadallah
OmniParser for Pure Vision Based GUI Agent
The recent success of large vision language models shows great potential in
driving the agent system operating on user interfaces. However, we argue that
the power multimodal models like GPT-4V as a general agent on multiple
operating systems across different applications is largely underestimated due
to the lack of a robust screen parsing technique capable of: 1) reliably
identifying interactable icons within the user interface, and 2) understanding
the semantics of various elements in a screenshot and accurately associate the
intended action with the corresponding region on the screen. To fill these
gaps, we introduce OmniParser, a comprehensive method for parsing user
interface screenshots into structured elements, which significantly enhances
the ability of GPT-4V to generate actions that can be accurately grounded in
the corresponding regions of the interface. We first curated an interactable
icon detection dataset using popular webpages and an icon description dataset.
These datasets were utilized to fine-tune specialized models: a detection model
to parse interactable regions on the screen and a caption model to extract the
functional semantics of the detected elements. OmniParser
significantly improves GPT-4V's performance on ScreenSpot benchmark. And on
Mind2Web and AITW benchmark, OmniParser with screenshot only input
outperforms the GPT-4V baselines requiring additional information outside of
screenshot.
OmniParser enhances GPT-4V's interaction with user interfaces by accurately parsing screenshots into actionable elements, significantly improving performance on key benchmarks.
vision language models multimodal models GPT-4V screen parsing interactable icons semantic elements detection model caption model ScreenSpot benchmark Mind2Web benchmark AITW benchmark