WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web
  Tasks
Powered by a large language model (LLM), a web browsing agent operates web
browsers in a human-like manner and offers a highly transparent path toward
automating a wide range of everyday tasks. As web agents become increasingly
capable and demonstrate proficiency in general browsing tasks, a critical
question emerges: Can they go beyond general browsing to robustly handle tasks
that are tedious and complex, or chores that humans often avoid doing
themselves? In this paper, we introduce WebChoreArena, a new fully reproducible
benchmark comprising 532 carefully curated tasks designed to extend the scope
of WebArena beyond general browsing to more labor-intensive and tedious tasks.
WebChoreArena systematically integrates three key challenges: (i) Massive
Memory tasks requiring accurate retrieval of large amounts of information in
the observations, (ii) Calculation tasks demanding precise mathematical
reasoning, and (iii) Long-Term Memory tasks necessitating long-term memory
across multiple webpages. Built on top of the fully reproducible and widely
adopted four WebArena simulation environments, WebChoreArena ensures strict
reproducibility and enables fair, direct comparisons with the established
WebArena benchmark, offering key insights into agent progress. Our experimental
results demonstrate that as LLMs evolve, represented by GPT-4o, Claude 3.7
Sonnet, and Gemini 2.5 Pro, significant improvements in performance are
observed on WebChoreArena. These findings suggest that WebChoreArena is
well-suited to measure the advancement of state-of-the-art LLMs with greater
clarity. Nevertheless, the results also indicate that even with Gemini 2.5 Pro,
there remains substantial room for improvement compared to WebArena,
highlighting the increased challenges posed by WebChoreArena.
683ebebfb5052f5f8741c7f5
Atsuyuki Miyai
claimed_verified
2025-06-03T09:39:31.731Z
6527b37c0ae663e384eb1b85
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png
Atsuyuki Miyai
user
AtsuMiyai
683ebebfb5052f5f8741c7f6
Zaiying Zhao
683ebebfb5052f5f8741c7f7
Kazuki Egashira
683ebebfb5052f5f8741c7f8
Atsuki Sato
683ebebfb5052f5f8741c7f9
Tatsumi Sunada
683ebebfb5052f5f8741c7fa
Shota Onohara
683ebebfb5052f5f8741c7fb
Hiromasa Yamanishi
683ebebfb5052f5f8741c7fc
Mashiro Toyooka
683ebebfb5052f5f8741c7fd
Kunato Nishina
683ebebfb5052f5f8741c7fe
Ryoma Maeda
683ebebfb5052f5f8741c7ff
Kiyoharu Aizawa
683ebebfb5052f5f8741c800
Toshihiko Yamasaki
WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web
  Tasks
Powered by a large language model (LLM), a web browsing agent operates web
browsers in a human-like manner and offers a highly transparent path toward
automating a wide range of everyday tasks. As web agents become increasingly
capable and demonstrate proficiency in general browsing tasks, a critical
question emerges: Can they go beyond general browsing to robustly handle tasks
that are tedious and complex, or chores that humans often avoid doing
themselves? In this paper, we introduce WebChoreArena, a new fully reproducible
benchmark comprising 532 carefully curated tasks designed to extend the scope
of WebArena beyond general browsing to more labor-intensive and tedious tasks.
WebChoreArena systematically integrates three key challenges: (i) Massive
Memory tasks requiring accurate retrieval of large amounts of information in
the observations, (ii) Calculation tasks demanding precise mathematical
reasoning, and (iii) Long-Term Memory tasks necessitating long-term memory
across multiple webpages. Built on top of the fully reproducible and widely
adopted four WebArena simulation environments, WebChoreArena ensures strict
reproducibility and enables fair, direct comparisons with the established
WebArena benchmark, offering key insights into agent progress. Our experimental
results demonstrate that as LLMs evolve, represented by GPT-4o, Claude 3.7
Sonnet, and Gemini 2.5 Pro, significant improvements in performance are
observed on WebChoreArena. These findings suggest that WebChoreArena is
well-suited to measure the advancement of state-of-the-art LLMs with greater
clarity. Nevertheless, the results also indicate that even with Gemini 2.5 Pro,
there remains substantial room for improvement compared to WebArena,
highlighting the increased challenges posed by WebChoreArena.
WebChoreArena, a new benchmark comprising 532 tasks, extends the scope of WebArena to more complex and tedious web browsing tasks, measuring advancements in LLM capabilities.
LLM Web browsing agent WebChoreArena benchmark general browsing Massive Memory tasks Calculation tasks Long-Term Memory tasks WebArena simulation environments GPT-4o Claude 3.7 Sonnet Gemini 2.5 Pro