WorldVLA: Towards Autoregressive Action World Model
We present WorldVLA, an autoregressive action world model that unifies action
and image understanding and generation. Our WorldVLA intergrates
Vision-Language-Action (VLA) model and world model in one single framework. The
world model predicts future images by leveraging both action and image
understanding, with the purpose of learning the underlying physics of the
environment to improve action generation. Meanwhile, the action model generates
the subsequent actions based on image observations, aiding in visual
understanding and in turn helps visual generation of the world model. We
demonstrate that WorldVLA outperforms standalone action and world models,
highlighting the mutual enhancement between the world model and the action
model. In addition, we find that the performance of the action model
deteriorates when generating sequences of actions in an autoregressive manner.
This phenomenon can be attributed to the model's limited generalization
capability for action prediction, leading to the propagation of errors from
earlier actions to subsequent ones. To address this issue, we propose an
attention mask strategy that selectively masks prior actions during the
generation of the current action, which shows significant performance
improvement in the action chunk generation task.
685e06f771131fa43be08abe
Jun Cen
claimed_verified
2025-07-03T16:19:12.188Z
644493bb9c1bd83bd1a09860
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VNb6fnFYraR5R4vSwMYZz.jpeg
Jun CEN
user
jcenaa
685e06f771131fa43be08abf
Chaohui Yu
685e06f771131fa43be08ac0
Hangjie Yuan
claimed_verified
2025-06-27T08:56:57.326Z
649d54b314afbb10ce2a9eeb
/avatars/15c325d8c2273ff63569f23015e98486.svg
Hangjie Yuan
user
JacobYuan
685e06f771131fa43be08ac1
Yuming Jiang
685e06f771131fa43be08ac2
Siteng Huang
685e06f771131fa43be08ac3
Jiayan Guo
685e06f771131fa43be08ac4
Xin Li
685e06f771131fa43be08ac5
Yibing Song
685e06f771131fa43be08ac6
Hao Luo
685e06f771131fa43be08ac7
Fan Wang
685e06f771131fa43be08ac8
Deli Zhao
685e06f771131fa43be08ac9
Hao Chen
WorldVLA: Towards Autoregressive Action World Model
We present WorldVLA, an autoregressive action world model that unifies action
and image understanding and generation. Our WorldVLA intergrates
Vision-Language-Action (VLA) model and world model in one single framework. The
world model predicts future images by leveraging both action and image
understanding, with the purpose of learning the underlying physics of the
environment to improve action generation. Meanwhile, the action model generates
the subsequent actions based on image observations, aiding in visual
understanding and in turn helps visual generation of the world model. We
demonstrate that WorldVLA outperforms standalone action and world models,
highlighting the mutual enhancement between the world model and the action
model. In addition, we find that the performance of the action model
deteriorates when generating sequences of actions in an autoregressive manner.
This phenomenon can be attributed to the model's limited generalization
capability for action prediction, leading to the propagation of errors from
earlier actions to subsequent ones. To address this issue, we propose an
attention mask strategy that selectively masks prior actions during the
generation of the current action, which shows significant performance
improvement in the action chunk generation task.
WorldVLA, an autoregressive action world model integrating vision-language-action (VLA) and world models, enhances performance through mutual understanding and generation, improving action prediction and sequence generation with an attention mask strategy.
autoregressive action world model Vision-Language-Action (VLA) model world model action generation action prediction attention mask strategy