Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation
  Model Internet Agents
The vision of a broadly capable and goal-directed agent, such as an
Internet-browsing agent in the digital world and a household humanoid in the
physical world, has rapidly advanced, thanks to the generalization capability
of foundation models. Such a generalist agent needs to have a large and diverse
skill repertoire, such as finding directions between two travel locations and
buying specific items from the Internet. If each skill needs to be specified
manually through a fixed set of human-annotated instructions, the agent's skill
repertoire will necessarily be limited due to the quantity and diversity of
human-annotated instructions. In this work, we address this challenge by
proposing Proposer-Agent-Evaluator, an effective learning system that enables
foundation model agents to autonomously discover and practice skills in the
wild. At the heart of PAE is a context-aware task proposer that autonomously
proposes tasks for the agent to practice with context information of the
environment such as user demos or even just the name of the website itself for
Internet-browsing agents. Then, the agent policy attempts those tasks with
thoughts and actual grounded operations in the real world with resulting
trajectories evaluated by an autonomous VLM-based success evaluator. The
success evaluation serves as the reward signal for the agent to refine its
policies through RL. We validate PAE on challenging vision-based web
navigation, using both real-world and self-hosted websites from WebVoyager and
WebArena.To the best of our knowledge, this work represents the first effective
learning system to apply autonomous task proposal with RL for agents that
generalizes real-world human-annotated benchmarks with SOTA performances. Our
open-source checkpoints and code can be found in https://yanqval.github.io/PAE/
6762d6291d6d92e318dfb0f4
Yifei Zhou
6762d6291d6d92e318dfb0f5
Qianlan Yang
extracted_confirmed
2024-12-18T18:06:53.583Z
662c11c4d64b8a3bee132130
/avatars/6c669c1d175bfc582e93f02c95b18f1e.svg
Qianlan Yang
user
yanQval
6762d6291d6d92e318dfb0f6
Kaixiang Lin
6762d6291d6d92e318dfb0f7
Min Bai
6762d6291d6d92e318dfb0f8
Xiong Zhou
6762d6291d6d92e318dfb0f9
Yu-Xiong Wang
6762d6291d6d92e318dfb0fa
Sergey Levine
6762d6291d6d92e318dfb0fb
Erran Li
Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation
  Model Internet Agents
The vision of a broadly capable and goal-directed agent, such as an
Internet-browsing agent in the digital world and a household humanoid in the
physical world, has rapidly advanced, thanks to the generalization capability
of foundation models. Such a generalist agent needs to have a large and diverse
skill repertoire, such as finding directions between two travel locations and
buying specific items from the Internet. If each skill needs to be specified
manually through a fixed set of human-annotated instructions, the agent's skill
repertoire will necessarily be limited due to the quantity and diversity of
human-annotated instructions. In this work, we address this challenge by
proposing Proposer-Agent-Evaluator, an effective learning system that enables
foundation model agents to autonomously discover and practice skills in the
wild. At the heart of PAE is a context-aware task proposer that autonomously
proposes tasks for the agent to practice with context information of the
environment such as user demos or even just the name of the website itself for
Internet-browsing agents. Then, the agent policy attempts those tasks with
thoughts and actual grounded operations in the real world with resulting
trajectories evaluated by an autonomous VLM-based success evaluator. The
success evaluation serves as the reward signal for the agent to refine its
policies through RL. We validate PAE on challenging vision-based web
navigation, using both real-world and self-hosted websites from WebVoyager and
WebArena.To the best of our knowledge, this work represents the first effective
learning system to apply autonomous task proposal with RL for agents that
generalizes real-world human-annotated benchmarks with SOTA performances. Our
open-source checkpoints and code can be found in https://yanqval.github.io/PAE/
A learning system (Proposer-Agent-Evaluator) enables foundation models to autonomously discover and practice skills by proposing tasks, attempting them, and being evaluated by a success evaluator using reinforcement learning.
Proposer-Agent-Evaluator context-aware task proposer agent policy VLM-based success evaluator reinforcement learning vision-based web navigation SOTA performances