WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum
  Reinforcement Learning
Large language models (LLMs) have shown remarkable potential as autonomous
agents, particularly in web-based tasks. However, existing LLM web agents
heavily rely on expensive proprietary LLM APIs, while open LLMs lack the
necessary decision-making capabilities. This paper introduces WebRL, a
self-evolving online curriculum reinforcement learning framework designed to
train high-performance web agents using open LLMs. WebRL addresses three key
challenges in building LLM web agents, including the scarcity of training
tasks, sparse feedback signals, and policy distribution drift in online
learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that
generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised
reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure
consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4
models into proficient web agents. On WebArena-Lite, WebRL improves the success
rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B.
These open models significantly surpass the performance of GPT-4-Turbo (17.6%)
and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained
on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's
effectiveness in bridging the gap between open and proprietary LLM-based web
agents, paving the way for more accessible and powerful autonomous web
interaction systems.
67299dd8ff3de35f30a2bffe
Zehan Qi
admin_assigned
2024-11-05T13:58:00.956Z
650ace9fa80a948447812fc5
/avatars/e01fe0d973008f5e3cd3c99f895b5946.svg
zehanqi
user
zehanqi
67299dd8ff3de35f30a2bfff
Xiao Liu
admin_assigned
2024-11-05T13:58:45.111Z
62f098602ca4d32a7cd87aba
https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
Xiao Liu
user
ShawLiu
67299dd8ff3de35f30a2c000
Iat Long Iong
67299dd8ff3de35f30a2c001
Hanyu Lai
admin_assigned
2024-11-05T13:56:47.720Z
62ea2d48b493272c269e8f34
/avatars/89b215dafa503b51ab212a9b63c82aca.svg
Hanyu Lai
user
hanyullai
67299dd8ff3de35f30a2c002
Xueqiao Sun
admin_assigned
2024-11-05T13:56:56.205Z
6729afc3c519b000f45d053b
/avatars/1d47cfd495e72316420fc16f7286df2b.svg
Xueqiao Sun
user
snow1007
67299dd8ff3de35f30a2c003
Xinyue Yang
67299dd8ff3de35f30a2c004
Jiadai Sun
admin_assigned
2024-11-05T13:57:27.980Z
64e82f2c9a3cd93b370e9936
/avatars/942e6008e8e78583c2750d678fcaadb1.svg
jiadaisun
user
jiadaisun
67299dd8ff3de35f30a2c005
Yu Yang
admin_assigned
2024-11-05T13:57:33.794Z
60a53adbf9b53404e7806277
https://cdn-avatars.huggingface.co/v1/production/uploads/1621441193991-noauth.jpeg
Yu Yang
user
yuyang
67299dd8ff3de35f30a2c006
Shuntian Yao
admin_assigned
2024-11-05T13:57:53.493Z
6464c91ef4f3102bb6c7418a
/avatars/1e6ce129d8cd51e81d517e23ad9aee43.svg
Shuntian Yao
user
TMIP
67299dd8ff3de35f30a2c007
Tianjie Zhang
claimed_verified
2024-11-05T07:57:23.819Z
64dceda0f44d2407e60179f8
/avatars/726d5ed318b40ae886b7387a532217e4.svg
Tianjie Zhang
user
tianjiezhang
67299dd8ff3de35f30a2c008
Wei Xu
67299dd8ff3de35f30a2c009
Jie Tang
67299dd8ff3de35f30a2c00a
Yuxiao Dong
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum
  Reinforcement Learning
Large language models (LLMs) have shown remarkable potential as autonomous
agents, particularly in web-based tasks. However, existing LLM web agents
heavily rely on expensive proprietary LLM APIs, while open LLMs lack the
necessary decision-making capabilities. This paper introduces WebRL, a
self-evolving online curriculum reinforcement learning framework designed to
train high-performance web agents using open LLMs. WebRL addresses three key
challenges in building LLM web agents, including the scarcity of training
tasks, sparse feedback signals, and policy distribution drift in online
learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that
generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised
reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure
consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4
models into proficient web agents. On WebArena-Lite, WebRL improves the success
rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B.
These open models significantly surpass the performance of GPT-4-Turbo (17.6%)
and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained
on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's
effectiveness in bridging the gap between open and proprietary LLM-based web
agents, paving the way for more accessible and powerful autonomous web
interaction systems.
WebRL enhances open LLMs as web agents through a self-evolving curriculum, robust outcome-supervised reward model, and adaptive reinforcement learning strategies, achieving high success rates compared to proprietary models.
reinforcement learning self-evolving curriculum outcome-supervised reward model adaptive reinforcement learning web agents WebRL Llama-3.1 GLM-4 WebArena-Lite GPT-4-Turbo GPT-4o AutoWebGLM