WebVoyager: Building an End-to-End Web Agent with Large Multimodal
  Models
The advancement of large language models (LLMs) leads to a new era marked by
the development of autonomous applications in the real world, which drives
innovation in the creation of advanced web-based agents. Existing web agents
typically only handle one input modality and are evaluated only in simplified
web simulators or static web snapshots, greatly limiting their applicability in
real-world scenarios. To bridge this gap, we introduce WebVoyager, an
innovative Large Multimodal Model (LMM) powered web agent that can complete
user instructions end-to-end by interacting with real-world websites. Moreover,
we propose a new evaluation protocol for web agents to address the challenges
of automatic evaluation of open-ended web agent tasks, leveraging the robust
multimodal comprehension capabilities of GPT-4V. We create a new benchmark by
gathering real-world tasks from 15 widely used websites to evaluate our agents.
We show that WebVoyager achieves a 55.7% task success rate, significantly
surpassing the performance of both GPT-4 (All Tools) and the WebVoyager
(text-only) setups, underscoring the exceptional capability of WebVoyager in
practical applications. We found that our proposed automatic evaluation
achieves 85.3% agreement with human judgment, paving the way for further
development of web agents in a real-world setting.
65b318d4dfa3789ef34877c4
Hongliang He
admin_assigned
2024-01-26T09:58:34.239Z
64e863d59c7e43f8a056a332
/avatars/44e411f06b70a9a030a452b8d5f8e663.svg
Hongliang He
user
Hongliang1997
65b318d4dfa3789ef34877c5
Wenlin Yao
admin_assigned
2024-01-26T09:58:41.216Z
634f18e4aae4bde2c8e2adca
/avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
Wenlin Yao
user
wenlinyao
65b318d4dfa3789ef34877c6
Kaixin Ma
admin_assigned
2024-01-26T09:58:52.341Z
64ae4f6280f308a395fd7c19
/avatars/5f1330f8187cd5e66aa517303659f110.svg
Kaixin Ma
user
kaixinm
65b318d4dfa3789ef34877c7
Wenhao Yu
admin_assigned
2024-01-26T09:59:25.043Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
65b318d4dfa3789ef34877c8
Yong Dai
65b318d4dfa3789ef34877c9
Hongming Zhang
65b318d4dfa3789ef34877ca
Zhenzhong Lan
admin_assigned
2024-01-26T10:00:19.736Z
62ce6dd785cfd21c04c7e6f5
/avatars/89837a5dea6e2d753d59caad142bed4a.svg
ZhenzhongLan
user
DannyLan
65b318d4dfa3789ef34877cb
Dong Yu
WebVoyager: Building an End-to-End Web Agent with Large Multimodal
  Models
The advancement of large language models (LLMs) leads to a new era marked by
the development of autonomous applications in the real world, which drives
innovation in the creation of advanced web-based agents. Existing web agents
typically only handle one input modality and are evaluated only in simplified
web simulators or static web snapshots, greatly limiting their applicability in
real-world scenarios. To bridge this gap, we introduce WebVoyager, an
innovative Large Multimodal Model (LMM) powered web agent that can complete
user instructions end-to-end by interacting with real-world websites. Moreover,
we propose a new evaluation protocol for web agents to address the challenges
of automatic evaluation of open-ended web agent tasks, leveraging the robust
multimodal comprehension capabilities of GPT-4V. We create a new benchmark by
gathering real-world tasks from 15 widely used websites to evaluate our agents.
We show that WebVoyager achieves a 55.7% task success rate, significantly
surpassing the performance of both GPT-4 (All Tools) and the WebVoyager
(text-only) setups, underscoring the exceptional capability of WebVoyager in
practical applications. We found that our proposed automatic evaluation
achieves 85.3% agreement with human judgment, paving the way for further
development of web agents in a real-world setting.
WebVoyager, a Large Multimodal Model, enhances web agent performance by handling multiple modalities and real-world web interactions, achieving high success rates in practical tasks.
Large Language Models Large Multimodal Model WebVoyager GPT-4V web agents multimodal comprehension task success rate automatic evaluation