AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Language agents, built on top of language models (LMs), are systems that can
interact with complex environments, such as the open web. In this work, we
examine whether such agents can perform realistic and time-consuming tasks on
the web, e.g., monitoring real-estate markets or locating relevant nearby
businesses. We introduce AssistantBench, a challenging new benchmark consisting
of 214 realistic tasks that can be automatically evaluated, covering different
scenarios and domains. We find that AssistantBench exposes the limitations of
current systems, including language models and retrieval-augmented language
models, as no model reaches an accuracy of more than 25 points. While
closed-book LMs perform well, they exhibit low precision since they tend to
hallucinate facts. State-of-the-art web agents reach a score of near zero.
Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly
outperforms previous agents, and an ensemble of SPA and closed-book models
reaches the best overall performance. Moreover, we analyze failures of current
systems and highlight that web navigation remains a major challenge.
669f26d53de1a03833d0db5e
Ori Yoran
claimed_verified
2024-07-23T07:47:23.933Z
610b7ccff5dd7bcefedaff57
/avatars/a1e77460105a56f84371fa02d530a71e.svg
Yoran
user
Ori
669f26d53de1a03833d0db5f
Samuel Joseph Amouyal
669f26d53de1a03833d0db60
Chaitanya Malaviya
claimed_verified
2024-07-23T08:01:24.934Z
646bad133e2a7b065948d4be
https://cdn-avatars.huggingface.co/v1/production/uploads/646bad133e2a7b065948d4be/Hk21DxumaBbkMJNZCKL4A.jpeg
Chaitanya Malaviya
user
cmalaviya
669f26d53de1a03833d0db61
Ben Bogin
669f26d53de1a03833d0db62
Ofir Press
669f26d53de1a03833d0db63
Jonathan Berant
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Language agents, built on top of language models (LMs), are systems that can
interact with complex environments, such as the open web. In this work, we
examine whether such agents can perform realistic and time-consuming tasks on
the web, e.g., monitoring real-estate markets or locating relevant nearby
businesses. We introduce AssistantBench, a challenging new benchmark consisting
of 214 realistic tasks that can be automatically evaluated, covering different
scenarios and domains. We find that AssistantBench exposes the limitations of
current systems, including language models and retrieval-augmented language
models, as no model reaches an accuracy of more than 25 points. While
closed-book LMs perform well, they exhibit low precision since they tend to
hallucinate facts. State-of-the-art web agents reach a score of near zero.
Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly
outperforms previous agents, and an ensemble of SPA and closed-book models
reaches the best overall performance. Moreover, we analyze failures of current
systems and highlight that web navigation remains a major challenge.
AssistantBench evaluates web agents on realistic tasks, revealing limitations of current systems, with SeePlanAct achieving the best performance and highlighting web navigation as a key challenge.
language models language agents web agents retrieval-augmented language models AssistantBench SeePlanAct web navigation