AgentNet: Open Foundations for Computer-Use Agents
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, LI PEIHANG, Fangyu Lei, Chen Wu, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Hu Jiarui, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Yiheng Xu, Danyang Zhang, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Y.Charles, Zhilin Yang, Tao Yu
Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed and proprietary. As these agents will increasingly mediate digital interactions and execute consequential decisions on our behalf, the research community needs access to truly open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose AgentNet, a comprehensive open-source framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AgentNet dataset, a dataset of 27K computer-use data samples spanning various operating systems, applications, and websites; (3) a pipeline that discretizes continuous actions into state-action pairs and synthesizes reflective long chain-of-thought (CoT) reasoning; (4) a training recipe for scalable CUA modeling; and (5) AgentNetBench, a multi-dimensional offline benchmark for faster CUA evaluation. Our AgentNet-7B, fine-tuned on AgentNet dataset, demonstrates strong performance on several CUA benchmarks, achieving a success rate of 20.1% on OSWorld and 21.1% on WindowsAgentArena. Our training recipe, particularly its advanced reasoning mechanisms and strategic data mixture, enables robust performance scaling with increased data size. Further in-depth analysis of our models also demonstrate strong cross-domain generalization and performance scaling with test-time compute. We will release the annotation tool, datasets, code, and models to build open foundations for further CUA research.