Watch and Learn: Learning to Use Computers from Online Videos
Computer use agents (CUAs) need to plan task workflows grounded in diverse,
ever-changing applications and environments, but learning is hindered by the
scarcity of large-scale, high-quality training data in the target application.
Existing datasets are domain-specific, static, and costly to annotate, while
current synthetic data generation methods often yield simplistic or misaligned
task demonstrations. To address these limitations, we introduce Watch & Learn
(W&L), a framework that converts human demonstration videos readily available
on the Internet into executable UI trajectories at scale. Instead of directly
generating trajectories or relying on ad hoc reasoning heuristics, we cast the
problem as an inverse dynamics objective: predicting the user's action from
consecutive screen states. This formulation reduces manual engineering, is
easier to learn, and generalizes more robustly across applications. Concretely,
we develop an inverse dynamics labeling pipeline with task-aware video
retrieval, generate over 53k high-quality trajectories from raw web videos, and
demonstrate that these trajectories improve CUAs both as in-context
demonstrations and as supervised training data. On the challenging OSWorld
benchmark, UI trajectories extracted with W&L consistently enhance both
general-purpose and state-of-the-art frameworks in-context, and deliver
stronger gains for open-source models under supervised training. These results
highlight web-scale human demonstration videos as a practical and scalable
foundation for advancing CUAs towards real-world deployment.
68e481d4e4e093a7044e4d59
Chan Hee Song
68e481d4e4e093a7044e4d5a
Yiwen Song
68e481d4e4e093a7044e4d5b
Palash Goyal
68e481d4e4e093a7044e4d5c
Yu Su
claimed_verified
2025-10-11T14:05:37.232Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
68e481d4e4e093a7044e4d5d
Oriana Riva
68e481d4e4e093a7044e4d5e
Hamid Palangi
68e481d4e4e093a7044e4d5f
Tomas Pfister
Watch and Learn: Learning to Use Computers from Online Videos
Computer use agents (CUAs) need to plan task workflows grounded in diverse,
ever-changing applications and environments, but learning is hindered by the
scarcity of large-scale, high-quality training data in the target application.
Existing datasets are domain-specific, static, and costly to annotate, while
current synthetic data generation methods often yield simplistic or misaligned
task demonstrations. To address these limitations, we introduce Watch & Learn
(W&L), a framework that converts human demonstration videos readily available
on the Internet into executable UI trajectories at scale. Instead of directly
generating trajectories or relying on ad hoc reasoning heuristics, we cast the
problem as an inverse dynamics objective: predicting the user's action from
consecutive screen states. This formulation reduces manual engineering, is
easier to learn, and generalizes more robustly across applications. Concretely,
we develop an inverse dynamics labeling pipeline with task-aware video
retrieval, generate over 53k high-quality trajectories from raw web videos, and
demonstrate that these trajectories improve CUAs both as in-context
demonstrations and as supervised training data. On the challenging OSWorld
benchmark, UI trajectories extracted with W&L consistently enhance both
general-purpose and state-of-the-art frameworks in-context, and deliver
stronger gains for open-source models under supervised training. These results
highlight web-scale human demonstration videos as a practical and scalable
foundation for advancing CUAs towards real-world deployment.
Watch & Learn converts web demonstration videos into UI trajectories to enhance computer use agents, improving both in-context demonstrations and supervised training.
inverse dynamics task-aware video retrieval UI trajectories OSWorld benchmark