GPT-4V(ision) is a Generalist Web Agent, if Grounded
The recent development on large multimodal models (LMMs), especially
GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
of multimodal models beyond traditional tasks like image captioning and visual
question answering. In this work, we explore the potential of LMMs like GPT-4V
as a generalist web agent that can follow natural language instructions to
complete tasks on any given website. We propose SEEACT, a generalist web agent
that harnesses the power of LMMs for integrated visual understanding and acting
on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
standard offline evaluation on cached websites, we enable a new online
evaluation setting by developing a tool that allows running web agents on live
websites. We show that GPT-4V presents a great potential for web agents - it
can successfully complete 50% of the tasks on live websites if we manually
ground its textual plans into actions on the websites. This substantially
outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
specifically fine-tuned for web agents. However, grounding still remains a
major challenge. Existing LMM grounding strategies like set-of-mark prompting
turns out not effective for web agents, and the best grounding strategy we
develop in this paper leverages both the HTML text and visuals. Yet, there is
still a substantial gap with oracle grounding, leaving ample room for further
improvement.
65960e11b02e572eb0bfd00c
Boyuan Zheng
claimed_verified
2024-03-18T08:37:29.919Z
631a95cfa66151e36e54a905
https://cdn-avatars.huggingface.co/v1/production/uploads/631a95cfa66151e36e54a905/jM_4vy9_Kd8zomFVYtsTo.png
Boyuan Zheng
user
boyuanzheng010
65960e11b02e572eb0bfd00d
Boyu Gou
claimed_verified
2024-01-04T09:35:11.800Z
6500870f1e14749e84f8f887
https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
Boyu Gou
user
BoyuNLP
65960e11b02e572eb0bfd00e
Jihyung Kil
65960e11b02e572eb0bfd00f
Huan Sun
65960e11b02e572eb0bfd010
Yu Su
claimed_verified
2024-01-04T09:35:14.490Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
GPT-4V(ision) is a Generalist Web Agent, if Grounded
The recent development on large multimodal models (LMMs), especially
GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
of multimodal models beyond traditional tasks like image captioning and visual
question answering. In this work, we explore the potential of LMMs like GPT-4V
as a generalist web agent that can follow natural language instructions to
complete tasks on any given website. We propose SEEACT, a generalist web agent
that harnesses the power of LMMs for integrated visual understanding and acting
on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
standard offline evaluation on cached websites, we enable a new online
evaluation setting by developing a tool that allows running web agents on live
websites. We show that GPT-4V presents a great potential for web agents - it
can successfully complete 50% of the tasks on live websites if we manually
ground its textual plans into actions on the websites. This substantially
outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
specifically fine-tuned for web agents. However, grounding still remains a
major challenge. Existing LMM grounding strategies like set-of-mark prompting
turns out not effective for web agents, and the best grounding strategy we
develop in this paper leverages both the HTML text and visuals. Yet, there is
still a substantial gap with oracle grounding, leaving ample room for further
improvement.
LMMs like GPT-4V demonstrate potential as generalist web agents by following natural language instructions to complete tasks on live websites, outperforming text-only LLMs and smaller models with a combination of HTML text and visuals for grounding, though challenges remain.
GPT-4V multimodal models SEEACT MIND2WEB benchmark web agents visual understanding GPT-4 FLAN-T5 BLIP-2 set-of-mark prompting oracle grounding