D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to
  Embodied AI
Large language models leverage internet-scale text data, yet embodied AI
remains constrained by the prohibitive costs of physical trajectory collection.
Desktop environments -- particularly gaming -- offer a compelling alternative:
they provide rich sensorimotor interactions at scale while maintaining the
structured observation-action coupling essential for embodied learning. We
present D2E (Desktop to Embodied AI), a framework that demonstrates desktop
interactions can serve as an effective pretraining substrate for robotics
embodied AI tasks. Unlike prior work that remained domain-specific (e.g., VPT
for Minecraft) or kept data proprietary (e.g., SIMA), D2E establishes a
complete pipeline from scalable desktop data collection to verified transfer in
embodied domains. Our framework comprises three components: (1) the OWA Toolkit
that unifies diverse desktop interactions into a standardized format with 152x
compression, (2) the Generalist-IDM that achieves strong zero-shot
generalization across unseen games through timestamp-based event prediction,
enabling internet-scale pseudo-labeling, and (3) VAPT that transfers
desktop-pretrained representations to physical manipulation and navigation.
Using 1.3K+ hours of data (259 hours of human demonstrations, and 1K+ hours of
pseudo-labeled gameplay), we achieve a total of 96.6% success rate on LIBERO
manipulation and 83.3% on CANVAS navigation benchmarks. This validates that
sensorimotor primitives in digital interactions exhibit sufficient invariance
to transfer meaningfully to physical embodied tasks, establishing desktop
pretraining as a practical paradigm for robotics. We will make all our work
public, including the OWA toolkit, datasets of human-collected and
pseudo-labeled, and VAPT-trained models available at
https://worv-ai.github.io/d2e/
68ec8eb7cd07fb414898ca8a
Suwhan Choi
claimed_verified
2025-10-15T03:14:17.589Z
632ced22ea6e62428aba1625
/avatars/0aae0ad956bdb2345732ca47fb28e3b6.svg
Suhwan Choi
user
MilkClouds
68ec8eb7cd07fb414898ca8b
Jaeyoon Jung
claimed_verified
2025-10-13T10:04:31.376Z
646484cfb90150b2706df03b
https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png
Jaeyoon Jung
user
lastdefiance20
68ec8eb7cd07fb414898ca8c
Haebin Seong
claimed_verified
2025-10-15T15:25:29.794Z
63a9379e2e05ca32e352d93b
/avatars/6cda37befc873a92ed6d5dcba507954a.svg
Haebin Seong
user
hbseong
68ec8eb7cd07fb414898ca8d
Minchan Kim
claimed_verified
2025-10-13T10:04:33.915Z
647046995b8225e31ae91359
/avatars/dd288377385efc7b8b24bc88bd9c27d6.svg
Minchan Kim
user
shovelingpig
68ec8eb7cd07fb414898ca8e
Minyeong Kim
68ec8eb7cd07fb414898ca8f
Yongjun Cho
claimed_verified
2025-10-13T16:09:03.297Z
65e972db5686ed1f5eff2327
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/er_WDGEq0HY8rpIg69v0G.png
Cho Yongjun
user
PurpleSand
68ec8eb7cd07fb414898ca90
Yoonshik Kim
68ec8eb7cd07fb414898ca91
Yubeen Park
68ec8eb7cd07fb414898ca92
Youngjae Yu
68ec8eb7cd07fb414898ca93
Yunsung Lee
claimed_verified
2025-10-13T10:04:28.653Z
63198bf1615c77c25d63e9ab
/avatars/6d591f87366e9990fed3c221dafdfae0.svg
Yunsung Lee
user
Maangeek
D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to
  Embodied AI
Large language models leverage internet-scale text data, yet embodied AI
remains constrained by the prohibitive costs of physical trajectory collection.
Desktop environments -- particularly gaming -- offer a compelling alternative:
they provide rich sensorimotor interactions at scale while maintaining the
structured observation-action coupling essential for embodied learning. We
present D2E (Desktop to Embodied AI), a framework that demonstrates desktop
interactions can serve as an effective pretraining substrate for robotics
embodied AI tasks. Unlike prior work that remained domain-specific (e.g., VPT
for Minecraft) or kept data proprietary (e.g., SIMA), D2E establishes a
complete pipeline from scalable desktop data collection to verified transfer in
embodied domains. Our framework comprises three components: (1) the OWA Toolkit
that unifies diverse desktop interactions into a standardized format with 152x
compression, (2) the Generalist-IDM that achieves strong zero-shot
generalization across unseen games through timestamp-based event prediction,
enabling internet-scale pseudo-labeling, and (3) VAPT that transfers
desktop-pretrained representations to physical manipulation and navigation.
Using 1.3K+ hours of data (259 hours of human demonstrations, and 1K+ hours of
pseudo-labeled gameplay), we achieve a total of 96.6% success rate on LIBERO
manipulation and 83.3% on CANVAS navigation benchmarks. This validates that
sensorimotor primitives in digital interactions exhibit sufficient invariance
to transfer meaningfully to physical embodied tasks, establishing desktop
pretraining as a practical paradigm for robotics. We will make all our work
public, including the OWA toolkit, datasets of human-collected and
pseudo-labeled, and VAPT-trained models available at
https://worv-ai.github.io/d2e/
D2E framework uses desktop interactions to pretrain embodied AI, achieving high success rates in physical manipulation and navigation tasks.
embodied AI desktop environments sensorimotor interactions OWA Toolkit Generalist-IDM timestamp-based event prediction zero-shot generalization internet-scale pseudo-labeling VAPT LIBERO manipulation CANVAS navigation desktop pretraining