How to Train Your LLM Web Agent: A Statistical Diagnosis
LLM-based web agents have recently made significant progress, but much of it
has occurred in closed-source systems, widening the gap with open-source
alternatives. Progress has been held back by two key challenges: first, a
narrow focus on single-step tasks that overlooks the complexity of multi-step
web interactions; and second, the high compute costs required to post-train
LLM-based web agents. To address this, we present the first statistically
grounded study on compute allocation for LLM web-agent post-training. Our
approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate
a Llama 3.3 70B teacher via supervised fine-tuning (SFT), followed by on-policy
reinforcement learning. We find this process highly sensitive to hyperparameter
choices, making exhaustive sweeps impractical. To spare others from expensive
trial-and-error, we sample 1,370 configurations and use bootstrapping to
estimate effective hyperparameters. Our results show that combining SFT with
on-policy RL consistently outperforms either approach alone on both WorkArena
and MiniWob++. Further, this strategy requires only 55% of the compute to match
the peak performance of pure SFT on MiniWob++, effectively pushing the
compute-performance Pareto frontier, and is the only strategy that can close
the gap with closed-source models.
686dbfe2cb5725779c60b314
Dheeraj Vattikonda
claimed_verified
2025-07-09T15:10:15.465Z
65fb7b1c876d4642e8340e95
/avatars/c258c4528315243b98c1e60ba398e98b.svg
Dheeraj Vattikonda
user
Dheeraj46329
686dbfe2cb5725779c60b315
Santhoshi Ravichandran
686dbfe2cb5725779c60b316
Emiliano Penaloza
claimed_verified
2025-07-09T14:41:04.559Z
66c4d0fd18743b108e4e260e
https://cdn-avatars.huggingface.co/v1/production/uploads/66c4d0fd18743b108e4e260e/xKLb9t6VZCxt464VCkQ0O.png
Emiliano Penaloza
user
ppEmiliano
686dbfe2cb5725779c60b317
Hadi Nekoei
686dbfe2cb5725779c60b318
Megh Thakkar
686dbfe2cb5725779c60b319
Thibault Le Sellier de Chezelles
686dbfe2cb5725779c60b31a
Nicolas Gontier
686dbfe2cb5725779c60b31b
Miguel Muñoz-Mármol
686dbfe2cb5725779c60b31c
Sahar Omidi Shayegan
686dbfe2cb5725779c60b31d
Stefania Raimondo
686dbfe2cb5725779c60b31e
Xue Liu
686dbfe2cb5725779c60b31f
Alexandre Drouin
686dbfe2cb5725779c60b320
Laurent Charlin
686dbfe2cb5725779c60b321
Alexandre Piché
686dbfe2cb5725779c60b322
Alexandre Lacoste
686dbfe2cb5725779c60b323
Massimo Caccia
How to Train Your LLM Web Agent: A Statistical Diagnosis
LLM-based web agents have recently made significant progress, but much of it
has occurred in closed-source systems, widening the gap with open-source
alternatives. Progress has been held back by two key challenges: first, a
narrow focus on single-step tasks that overlooks the complexity of multi-step
web interactions; and second, the high compute costs required to post-train
LLM-based web agents. To address this, we present the first statistically
grounded study on compute allocation for LLM web-agent post-training. Our
approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate
a Llama 3.3 70B teacher via supervised fine-tuning (SFT), followed by on-policy
reinforcement learning. We find this process highly sensitive to hyperparameter
choices, making exhaustive sweeps impractical. To spare others from expensive
trial-and-error, we sample 1,370 configurations and use bootstrapping to
estimate effective hyperparameters. Our results show that combining SFT with
on-policy RL consistently outperforms either approach alone on both WorkArena
and MiniWob++. Further, this strategy requires only 55% of the compute to match
the peak performance of pure SFT on MiniWob++, effectively pushing the
compute-performance Pareto frontier, and is the only strategy that can close
the gap with closed-source models.
A study on compute allocation for post-training LLM-based web agents finds that combining supervised fine-tuning with on-policy reinforcement learning improves performance and reduces computational costs compared to using either method alone.
LLM-based web agents supervised fine-tuning on-policy reinforcement learning hyperparameter choices bootstrapping WorkArena MiniWob++