Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Web navigation is a unique domain that can automate many repetitive real-life
tasks and is challenging as it requires long-horizon sequential decision making
beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
reward models for web navigation that can be utilized during both training and
test-time have been absent until now. Despite the importance of speed and
cost-effectiveness, prior works have utilized MLLMs as reward models, which
poses significant constraints for real-world deployment. To address this, in
this work, we propose the first process reward model (PRM) called Web-Shepherd
which could assess web navigation trajectories in a step-level. To achieve
this, we first construct the WebPRM Collection, a large-scale dataset with 40K
step-level preference pairs and annotated checklists spanning diverse domains
and difficulty levels. Next, we also introduce the WebRewardBench, the first
meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
that our Web-Shepherd achieves about 30 points better accuracy compared to
using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
as the verifier. Our model, dataset, and code are publicly available at LINK.
682e854551706f69070aca6b
Hyungjoo Chae
claimed_verified
2025-05-22T07:16:37.301Z
64c8f4cec547ed5243ebd0a8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8f4cec547ed5243ebd0a8/MiOH5YbMg8Gh9KYlQsLmX.jpeg
Hyungjoo Chae
user
hyungjoochae
682e854551706f69070aca6c
Sunghwan Kim
claimed_verified
2025-05-22T07:16:32.322Z
646a0897c37ca1e12308b026
/avatars/6d720a9e366db9bec15c8c10878c0c75.svg
Sunghwan Kim
user
KimSHine
682e854551706f69070aca6d
Junhee Cho
682e854551706f69070aca6e
Seungone Kim
claimed_verified
2025-05-27T07:55:27.225Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
682e854551706f69070aca6f
Seungjun Moon
682e854551706f69070aca70
Gyeom Hwangbo
682e854551706f69070aca71
Dongha Lim
claimed_verified
2025-05-22T07:16:34.741Z
6683b8680b72be136701de35
/avatars/0c135e570b16b81ee2fb81ad65b01ba8.svg
Dongha Lim
user
donghalim
682e854551706f69070aca72
Minjin Kim
682e854551706f69070aca73
Yeonjun Hwang
682e854551706f69070aca74
Minju Gwak
682e854551706f69070aca75
Dongwook Choi
682e854551706f69070aca76
Minseok Kang
682e854551706f69070aca77
Gwanhoon Im
682e854551706f69070aca78
ByeongUng Cho
682e854551706f69070aca79
Hyojun Kim
682e854551706f69070aca7a
Jun Hee Han
682e854551706f69070aca7b
Taeyoon Kwon
claimed_verified
2025-05-27T07:55:29.445Z
636b529ef796304dd67d139c
/avatars/7a64d5095fcb1da558b52ad48177ad76.svg
Taeyoon Kwon
user
Connoriginal
682e854551706f69070aca7c
Minju Kim
682e854551706f69070aca7d
Beong-woo Kwak
682e854551706f69070aca7e
Dongjin Kang
682e854551706f69070aca7f
Jinyoung Yeo
Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Web navigation is a unique domain that can automate many repetitive real-life
tasks and is challenging as it requires long-horizon sequential decision making
beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
reward models for web navigation that can be utilized during both training and
test-time have been absent until now. Despite the importance of speed and
cost-effectiveness, prior works have utilized MLLMs as reward models, which
poses significant constraints for real-world deployment. To address this, in
this work, we propose the first process reward model (PRM) called Web-Shepherd
which could assess web navigation trajectories in a step-level. To achieve
this, we first construct the WebPRM Collection, a large-scale dataset with 40K
step-level preference pairs and annotated checklists spanning diverse domains
and difficulty levels. Next, we also introduce the WebRewardBench, the first
meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
that our Web-Shepherd achieves about 30 points better accuracy compared to
using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
as the verifier. Our model, dataset, and code are publicly available at LINK.
The paper introduces Web-Shepherd, a process reward model for web navigation, which improves accuracy and cost-effectiveness in step-level trajectory assessment compared to existing multimodal large language models.
multimodal large language model process reward model web navigation webPRM collection webrewardbench long-horizon sequential decision making preference pairs annotated checklists step-level assessment webarena-lite policy verifier