SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from
  Experience
Repurposing large vision-language models (LVLMs) as computer use agents
(CUAs) has led to substantial breakthroughs, primarily driven by human-labeled
data. However, these models often struggle with novel and specialized software,
particularly in scenarios lacking human annotations. To address this challenge,
we propose SEAgent, an agentic self-evolving framework enabling CUAs to
autonomously evolve through interactions with unfamiliar software.
Specifically, SEAgent empowers computer-use agents to autonomously master novel
software environments via experiential learning, where agents explore new
software, learn through iterative trial-and-error, and progressively tackle
auto-generated tasks organized from simple to complex. To achieve this goal, we
design a World State Model for step-wise trajectory assessment, along with a
Curriculum Generator that generates increasingly diverse and challenging tasks.
The agent's policy is updated through experiential learning, comprised of
adversarial imitation of failure actions and Group Relative Policy Optimization
(GRPO) on successful ones. Furthermore, we introduce a specialist-to-generalist
training strategy that integrates individual experiential insights from
specialist agents, facilitating the development of a stronger generalist CUA
capable of continuous autonomous evolution. This unified agent ultimately
achieves performance surpassing ensembles of individual specialist agents on
their specialized software. We validate the effectiveness of SEAgent across
five novel software environments within OS-World. Our approach achieves a
significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a
competitive open-source CUA, i.e., UI-TARS.
68941f54741a16f544fbcf0f
Zeyi Sun
claimed_verified
2025-08-07T10:38:28.544Z
63fda3fced9eead590ff6918
https://cdn-avatars.huggingface.co/v1/production/uploads/1677566802735-noauth.jpeg
Zeyi Sun
user
Zery
68941f54741a16f544fbcf10
Ziyu Liu
68941f54741a16f544fbcf11
Yuhang Zang
claimed_verified
2025-08-07T10:38:25.677Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
68941f54741a16f544fbcf12
Yuhang Cao
68941f54741a16f544fbcf13
Xiaoyi Dong
claimed_verified
2025-08-08T16:21:55.050Z
68943a6e8d3fb6db77ce2874
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7LBz6WhQmiKxOqZByJhr9.jpeg
Xiaoyi Dong
user
LightDong
68941f54741a16f544fbcf14
Tong Wu
68941f54741a16f544fbcf15
Dahua Lin
68941f54741a16f544fbcf16
Jiaqi Wang
claimed_verified
2025-08-14T13:40:05.351Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from
  Experience
Repurposing large vision-language models (LVLMs) as computer use agents
(CUAs) has led to substantial breakthroughs, primarily driven by human-labeled
data. However, these models often struggle with novel and specialized software,
particularly in scenarios lacking human annotations. To address this challenge,
we propose SEAgent, an agentic self-evolving framework enabling CUAs to
autonomously evolve through interactions with unfamiliar software.
Specifically, SEAgent empowers computer-use agents to autonomously master novel
software environments via experiential learning, where agents explore new
software, learn through iterative trial-and-error, and progressively tackle
auto-generated tasks organized from simple to complex. To achieve this goal, we
design a World State Model for step-wise trajectory assessment, along with a
Curriculum Generator that generates increasingly diverse and challenging tasks.
The agent's policy is updated through experiential learning, comprised of
adversarial imitation of failure actions and Group Relative Policy Optimization
(GRPO) on successful ones. Furthermore, we introduce a specialist-to-generalist
training strategy that integrates individual experiential insights from
specialist agents, facilitating the development of a stronger generalist CUA
capable of continuous autonomous evolution. This unified agent ultimately
achieves performance surpassing ensembles of individual specialist agents on
their specialized software. We validate the effectiveness of SEAgent across
five novel software environments within OS-World. Our approach achieves a
significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a
competitive open-source CUA, i.e., UI-TARS.
SEAgent, an agentic self-evolving framework, enables computer-use agents to autonomously master novel software through experiential learning and a curriculum of tasks, achieving superior performance compared to existing methods.
vision-language models computer use agents SEAgent experiential learning World State Model Curriculum Generator adversarial imitation Group Relative Policy Optimization specialist-to-generalist training OS-World UI-TARS