PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital
  World
Imagine a world where AI can handle your work while you sleep - organizing
your research materials, drafting a report, or creating a presentation you need
for tomorrow. However, while current digital agents can perform simple tasks,
they are far from capable of handling the complex real-world work that humans
routinely perform. We present PC Agent, an AI system that demonstrates a
crucial step toward this vision through human cognition transfer. Our key
insight is that the path from executing simple "tasks" to handling complex
"work" lies in efficiently capturing and learning from human cognitive
processes during computer use. To validate this hypothesis, we introduce three
key innovations: (1) PC Tracker, a lightweight infrastructure that efficiently
collects high-quality human-computer interaction trajectories with complete
cognitive context; (2) a two-stage cognition completion pipeline that
transforms raw interaction data into rich cognitive trajectories by completing
action semantics and thought processes; and (3) a multi-agent system combining
a planning agent for decision-making with a grounding agent for robust visual
grounding. Our preliminary experiments in PowerPoint presentation creation
reveal that complex digital work capabilities can be achieved with a small
amount of high-quality cognitive data - PC Agent, trained on just 133 cognitive
trajectories, can handle sophisticated work scenarios involving up to 50 steps
across multiple applications. This demonstrates the data efficiency of our
approach, highlighting that the key to training capable digital agents lies in
collecting human cognitive data. By open-sourcing our complete framework,
including the data collection infrastructure and cognition completion methods,
we aim to lower the barriers for the research community to develop truly
capable digital agents.
676a8186b88f71d971444400
Yanheng He
claimed_verified
2024-12-30T19:34:38.504Z
661b9ac57cfb7bcb3057a578
/avatars/f8afaa8eaad3a1e5963a4feebec3f7ab.svg
Yanheng He
user
henryhe0123
676a8186b88f71d971444401
Jiahe Jin
claimed_verified
2025-05-20T07:23:30.794Z
663f5e959e6f865ec6d4fb62
/avatars/eb0a908562b2c57335ae8bb949220430.svg
Jiahe Jin
user
zizi-0123
676a8186b88f71d971444402
Shijie Xia
claimed_verified
2025-02-06T14:15:31.725Z
65900d4ff5a209eeac08b463
https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
shijie xia
user
seven-cat
676a8186b88f71d971444403
Jiadi Su
676a8186b88f71d971444404
Runze Fan
claimed_verified
2024-12-30T19:34:41.541Z
616bfc2b40e2f69baa1c7add
https://cdn-avatars.huggingface.co/v1/production/uploads/616bfc2b40e2f69baa1c7add/Os7_qgMei-2lRVelrOG7B.jpeg
Run-Ze Fan
user
Vfrz
676a8186b88f71d971444405
Haoyang Zou
676a8186b88f71d971444406
Xiangkun Hu
676a8186b88f71d971444407
Pengfei Liu
claimed_verified
2025-06-29T10:14:54.589Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital
  World
Imagine a world where AI can handle your work while you sleep - organizing
your research materials, drafting a report, or creating a presentation you need
for tomorrow. However, while current digital agents can perform simple tasks,
they are far from capable of handling the complex real-world work that humans
routinely perform. We present PC Agent, an AI system that demonstrates a
crucial step toward this vision through human cognition transfer. Our key
insight is that the path from executing simple "tasks" to handling complex
"work" lies in efficiently capturing and learning from human cognitive
processes during computer use. To validate this hypothesis, we introduce three
key innovations: (1) PC Tracker, a lightweight infrastructure that efficiently
collects high-quality human-computer interaction trajectories with complete
cognitive context; (2) a two-stage cognition completion pipeline that
transforms raw interaction data into rich cognitive trajectories by completing
action semantics and thought processes; and (3) a multi-agent system combining
a planning agent for decision-making with a grounding agent for robust visual
grounding. Our preliminary experiments in PowerPoint presentation creation
reveal that complex digital work capabilities can be achieved with a small
amount of high-quality cognitive data - PC Agent, trained on just 133 cognitive
trajectories, can handle sophisticated work scenarios involving up to 50 steps
across multiple applications. This demonstrates the data efficiency of our
approach, highlighting that the key to training capable digital agents lies in
collecting human cognitive data. By open-sourcing our complete framework,
including the data collection infrastructure and cognition completion methods,
we aim to lower the barriers for the research community to develop truly
capable digital agents.
The PC Agent system demonstrates digital work capabilities through human cognition transfer using a PC Tracker, cognition completion pipeline, and multi-agent system, showing data efficiency in training digital agents.
PC Agent human cognition transfer PC Tracker cognition completion pipeline planning agent grounding agent multi-agent system