Explorer: Scaling Exploration-driven Web Trajectory Synthesis for
  Multimodal Web Agents
Recent success in large multimodal models (LMMs) has sparked promising
applications of agents capable of autonomously completing complex web tasks.
While open-source LMM agents have made significant advances in offline
evaluation benchmarks, their performance still falls substantially short of
human-level capabilities in more realistic online settings. A key bottleneck is
the lack of diverse and large-scale trajectory-level datasets across various
domains, which are expensive to collect. In this paper, we address this
challenge by developing a scalable recipe to synthesize the largest and most
diverse trajectory-level dataset to date, containing over 94K successful
multimodal web trajectories, spanning 49K unique URLs, 720K screenshots, and
33M web elements. In particular, we leverage extensive web exploration and
refinement to obtain diverse task intents. The average cost is 28 cents per
successful trajectory, making it affordable to a wide range of users in the
community. Leveraging this dataset, we train Explorer, a multimodal web agent,
and demonstrate strong performance on both offline and online web agent
benchmarks such as Mind2Web-Live, Multimodal-Mind2Web, and MiniWob++.
Additionally, our experiments highlight data scaling as a key driver for
improving web agent capabilities. We hope this study makes state-of-the-art
LMM-based agent research at a larger scale more accessible.
67b3f1f1f5bd60d66133e1f3
Vardaan Pahuja
claimed_verified
2025-02-18T09:31:47.969Z
6556717676fe5cfa6a115405
/avatars/570dd8f4eb6baaff12d7ebe11dde6348.svg
Vardaan Pahuja
user
vardaan123
67b3f1f1f5bd60d66133e1f4
Yadong Lu
extracted_pending
2025-02-18T02:35:29.988Z
664bbd75a6bd1b3d2ac7fc34
/avatars/127bf5d611b46ef95a1859a8cf21a160.svg
Yadong Lu
user
adamlu1
67b3f1f1f5bd60d66133e1f5
Corby Rosset
67b3f1f1f5bd60d66133e1f6
Boyu Gou
claimed_verified
2025-02-20T17:33:00.709Z
6500870f1e14749e84f8f887
https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
Boyu Gou
user
BoyuNLP
67b3f1f1f5bd60d66133e1f7
Arindam Mitra
67b3f1f1f5bd60d66133e1f8
Spencer Whitehead
67b3f1f1f5bd60d66133e1f9
Yu Su
67b3f1f1f5bd60d66133e1fa
Ahmed Awadallah
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for
  Multimodal Web Agents
Recent success in large multimodal models (LMMs) has sparked promising
applications of agents capable of autonomously completing complex web tasks.
While open-source LMM agents have made significant advances in offline
evaluation benchmarks, their performance still falls substantially short of
human-level capabilities in more realistic online settings. A key bottleneck is
the lack of diverse and large-scale trajectory-level datasets across various
domains, which are expensive to collect. In this paper, we address this
challenge by developing a scalable recipe to synthesize the largest and most
diverse trajectory-level dataset to date, containing over 94K successful
multimodal web trajectories, spanning 49K unique URLs, 720K screenshots, and
33M web elements. In particular, we leverage extensive web exploration and
refinement to obtain diverse task intents. The average cost is 28 cents per
successful trajectory, making it affordable to a wide range of users in the
community. Leveraging this dataset, we train Explorer, a multimodal web agent,
and demonstrate strong performance on both offline and online web agent
benchmarks such as Mind2Web-Live, Multimodal-Mind2Web, and MiniWob++.
Additionally, our experiments highlight data scaling as a key driver for
improving web agent capabilities. We hope this study makes state-of-the-art
LMM-based agent research at a larger scale more accessible.
A scalable dataset generates diverse multimodal web trajectories to enhance performance of LMM-based web agents.
multimodal models LMMs trajectory-level datasets web exploration multimodal web agent web agent benchmarks Mind2Web-Live Multimodal-Mind2Web MiniWob++