Latent Action Pretraining from Videos
We introduce Latent Action Pretraining for general Action models (LAPA), an
unsupervised method for pretraining Vision-Language-Action (VLA) models without
ground-truth robot action labels. Existing Vision-Language-Action models
require action labels typically collected by human teleoperators during
pretraining, which significantly limits possible data sources and scale. In
this work, we propose a method to learn from internet-scale videos that do not
have robot action labels. We first train an action quantization model
leveraging VQ-VAE-based objective to learn discrete latent actions between
image frames, then pretrain a latent VLA model to predict these latent actions
from observations and task descriptions, and finally finetune the VLA on
small-scale robot manipulation data to map from latent to robot actions.
Experimental results demonstrate that our method significantly outperforms
existing techniques that train robot manipulation policies from large-scale
videos. Furthermore, it outperforms the state-of-the-art VLA model trained with
robotic action labels on real-world manipulation tasks that require language
conditioning, generalization to unseen objects, and semantic generalization to
unseen instructions. Training only on human manipulation videos also shows
positive transfer, opening up the potential for leveraging web-scale data for
robotics foundation model.
670f15d6cdc5f5a980175c6a
Seonghyeon Ye
670f15d6cdc5f5a980175c6b
Joel Jang
670f15d6cdc5f5a980175c6c
Byeongguk Jeon
670f15d6cdc5f5a980175c6d
Sejune Joo
670f15d6cdc5f5a980175c6e
Jianwei Yang
670f15d6cdc5f5a980175c6f
Baolin Peng
670f15d6cdc5f5a980175c70
Ajay Mandlekar
670f15d6cdc5f5a980175c71
Reuben Tan
670f15d6cdc5f5a980175c72
Yu-Wei Chao
670f15d6cdc5f5a980175c73
Bill Yuchen Lin
670f15d6cdc5f5a980175c74
Lars Liden
claimed_verified
2025-02-21T16:15:28.270Z
65bc13c7719492167d777718
/avatars/d1a3dc52c130b84a47c8a4ddd2e74be8.svg
Lars Liden
user
larsliden
670f15d6cdc5f5a980175c75
Kimin Lee
670f15d6cdc5f5a980175c76
Jianfeng Gao
670f15d6cdc5f5a980175c77
Luke Zettlemoyer
670f15d6cdc5f5a980175c78
Dieter Fox
670f15d6cdc5f5a980175c79
Minjoon Seo
Latent Action Pretraining from Videos
We introduce Latent Action Pretraining for general Action models (LAPA), an
unsupervised method for pretraining Vision-Language-Action (VLA) models without
ground-truth robot action labels. Existing Vision-Language-Action models
require action labels typically collected by human teleoperators during
pretraining, which significantly limits possible data sources and scale. In
this work, we propose a method to learn from internet-scale videos that do not
have robot action labels. We first train an action quantization model
leveraging VQ-VAE-based objective to learn discrete latent actions between
image frames, then pretrain a latent VLA model to predict these latent actions
from observations and task descriptions, and finally finetune the VLA on
small-scale robot manipulation data to map from latent to robot actions.
Experimental results demonstrate that our method significantly outperforms
existing techniques that train robot manipulation policies from large-scale
videos. Furthermore, it outperforms the state-of-the-art VLA model trained with
robotic action labels on real-world manipulation tasks that require language
conditioning, generalization to unseen objects, and semantic generalization to
unseen instructions. Training only on human manipulation videos also shows
positive transfer, opening up the potential for leveraging web-scale data for
robotics foundation model.
Latent Action Pretraining (LAPA) enables unsupervised learning of Vision-Language-Action (VLA) models using internet-scale human videos without action labels, achieving superior performance on robotics manipulation tasks.
Vision-Language-Action LAPA action quantization VQ-VAE latent actions semantic generalization