VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
Training computer-use agents requires massive amounts of GUI interaction
data, but manually annotating action trajectories at scale is prohibitively
expensive. We present VideoAgentTrek, a scalable pipeline that automatically
mines training data from publicly available screen-recorded videos at web
scale, eliminating the need for manual annotation. Our approach addresses a key
challenge: raw videos contain implicit demonstrations but lack explicit action
labels. To solve this, we develop Video2Action, an inverse dynamics module
(IDM) with two components: (1) a video grounding model that detects and
localizes GUI actions with precise temporal boundaries and context, and (2) an
action-content recognizer that extracts structured parameters like click
coordinates and typed text with high fidelity. Applied to 39,000 YouTube
tutorial videos, our pipeline generates 1.52 million interaction steps
automatically. We leverage this data through continued pretraining followed by
supervised fine-tuning. On OSWorld-Verified, our approach improves task success
rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On
AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results
demonstrate that passive internet videos can be transformed into high-quality
supervision for computer-use agents, providing a scalable alternative to
expensive manual annotation.
68f98a91b9b2e4ae0467376d
Dunjie Lu
claimed_verified
2025-10-23T14:50:39.730Z
669ca7e678115e16bdfc9bfc
https://cdn-avatars.huggingface.co/v1/production/uploads/669ca7e678115e16bdfc9bfc/pku8NvQKqfNQACRqm1YrW.jpeg
Lu Dunjie
user
ludunjie
68f98a91b9b2e4ae0467376e
Yiheng Xu
68f98a91b9b2e4ae0467376f
Junli Wang
68f98a91b9b2e4ae04673770
Haoyuan Wu
68f98a91b9b2e4ae04673771
Xinyuan Wang
68f98a91b9b2e4ae04673772
Zekun Wang
68f98a91b9b2e4ae04673773
Junlin Yang
68f98a91b9b2e4ae04673774
Hongjin Su
68f98a91b9b2e4ae04673775
Jixuan Chen
68f98a91b9b2e4ae04673776
Junda Chen
68f98a91b9b2e4ae04673777
Yuchen Mao
68f98a91b9b2e4ae04673778
Jingren Zhou
68f98a91b9b2e4ae04673779
Junyang Lin
68f98a91b9b2e4ae0467377a
Binyuan Hui
claimed_verified
2025-10-23T14:50:33.160Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
68f98a91b9b2e4ae0467377b
Tao Yu
VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
Training computer-use agents requires massive amounts of GUI interaction
data, but manually annotating action trajectories at scale is prohibitively
expensive. We present VideoAgentTrek, a scalable pipeline that automatically
mines training data from publicly available screen-recorded videos at web
scale, eliminating the need for manual annotation. Our approach addresses a key
challenge: raw videos contain implicit demonstrations but lack explicit action
labels. To solve this, we develop Video2Action, an inverse dynamics module
(IDM) with two components: (1) a video grounding model that detects and
localizes GUI actions with precise temporal boundaries and context, and (2) an
action-content recognizer that extracts structured parameters like click
coordinates and typed text with high fidelity. Applied to 39,000 YouTube
tutorial videos, our pipeline generates 1.52 million interaction steps
automatically. We leverage this data through continued pretraining followed by
supervised fine-tuning. On OSWorld-Verified, our approach improves task success
rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On
AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results
demonstrate that passive internet videos can be transformed into high-quality
supervision for computer-use agents, providing a scalable alternative to
expensive manual annotation.
VideoAgentTrek automatically extracts GUI interaction data from YouTube videos using Video2Action, an inverse dynamics module, improving task success rates and step accuracy for computer-use agents.
inverse dynamics module video grounding model action-content recognizer continued pretraining supervised fine-tuning OSWorld-Verified AgentNetBench