RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
This paper presents RynnVLA-001, a vision-language-action(VLA) model built
upon large-scale video generative pretraining from human demonstrations. We
propose a novel two-stage pretraining methodology. The first stage, Ego-Centric
Video Generative Pretraining, trains an Image-to-Video model on 12M ego-centric
manipulation videos to predict future frames conditioned on an initial frame
and a language instruction. The second stage, Human-Centric Trajectory-Aware
Modeling, extends this by jointly predicting future keypoint trajectories,
thereby effectively bridging visual frame prediction with action prediction.
Furthermore, to enhance action representation, we propose ActionVAE, a
variational autoencoder that compresses sequences of actions into compact
latent embeddings, reducing the complexity of the VLA output space. When
finetuned on the same downstream robotics datasets, RynnVLA-001 achieves
superior performance over state-of-the-art baselines, demonstrating that the
proposed pretraining strategy provides a more effective initialization for VLA
models.
68ccb5443df9ac65e93dc5e8
Yuming Jiang
admin_assigned
2025-09-19T10:43:28.955Z
629c95b7a5d6f5fe10e6ed45
https://cdn-avatars.huggingface.co/v1/production/uploads/629c95b7a5d6f5fe10e6ed45/Sy0Ype5snsRookID-gsSm.jpeg
Yuming Jiang
user
yumingj
68ccb5443df9ac65e93dc5e9
Siteng Huang
admin_assigned
2025-09-19T10:43:37.080Z
65fd82762bf2cd20ddaa193f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
Siteng Huang
user
huangsiteng
68ccb5443df9ac65e93dc5ea
Shengke Xue
68ccb5443df9ac65e93dc5eb
Yaxi Zhao
admin_assigned
2025-09-19T10:44:49.332Z
676124d3b8b1b60a0e5c2d8e
/avatars/350e7b9fb6f171cea7dafa39d3890cb5.svg
Yaxi Zhao
user
yaniii
68ccb5443df9ac65e93dc5ec
Jun Cen
68ccb5443df9ac65e93dc5ed
Sicong Leng
admin_assigned
2025-09-19T10:45:11.063Z
609115c79a8bcaa437b234a9
/avatars/1631a91030703d8397133363cf82c863.svg
Leng Sicong
user
Sicong
68ccb5443df9ac65e93dc5ee
Kehan Li
68ccb5443df9ac65e93dc5ef
Jiayan Guo
admin_assigned
2025-09-19T10:45:19.380Z
66224557c61c7fbd98099079
/avatars/a4f2144585c808865c73b5b7f0087c1f.svg
Jiayan Guo
user
SpaceProduct
68ccb5443df9ac65e93dc5f0
Kexiang Wang
68ccb5443df9ac65e93dc5f1
Mingxiu Chen
68ccb5443df9ac65e93dc5f2
Fan Wang
68ccb5443df9ac65e93dc5f3
Deli Zhao
68ccb5443df9ac65e93dc5f4
Xin Li
RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
This paper presents RynnVLA-001, a vision-language-action(VLA) model built
upon large-scale video generative pretraining from human demonstrations. We
propose a novel two-stage pretraining methodology. The first stage, Ego-Centric
Video Generative Pretraining, trains an Image-to-Video model on 12M ego-centric
manipulation videos to predict future frames conditioned on an initial frame
and a language instruction. The second stage, Human-Centric Trajectory-Aware
Modeling, extends this by jointly predicting future keypoint trajectories,
thereby effectively bridging visual frame prediction with action prediction.
Furthermore, to enhance action representation, we propose ActionVAE, a
variational autoencoder that compresses sequences of actions into compact
latent embeddings, reducing the complexity of the VLA output space. When
finetuned on the same downstream robotics datasets, RynnVLA-001 achieves
superior performance over state-of-the-art baselines, demonstrating that the
proposed pretraining strategy provides a more effective initialization for VLA
models.
RynnVLA-001, a vision-language-action model, uses a two-stage pretraining approach and ActionVAE to achieve superior performance on robotics tasks.
Ego-Centric Video Generative Pretraining Human-Centric Trajectory-Aware Modeling Image-to-Video model keypoint trajectories ActionVAE variational autoencoder latent embeddings VLA model