DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive
  World Knowledge
Recent advances in vision-language-action (VLA) models have shown promise in
integrating image generation with action prediction to improve generalization
and reasoning in robot manipulation. However, existing methods are limited to
challenging image-based forecasting, which suffers from redundant information
and lacks comprehensive and critical world knowledge, including dynamic,
spatial and semantic information. To address these limitations, we propose
DreamVLA, a novel VLA framework that integrates comprehensive world knowledge
forecasting to enable inverse dynamics modeling, thereby establishing a
perception-prediction-action loop for manipulation tasks. Specifically,
DreamVLA introduces a dynamic-region-guided world knowledge prediction,
integrated with the spatial and semantic cues, which provide compact yet
comprehensive representations for action planning. This design aligns with how
humans interact with the world by first forming abstract multimodal reasoning
chains before acting. To mitigate interference among the dynamic, spatial and
semantic information during training, we adopt a block-wise structured
attention mechanism that masks their mutual attention, preventing information
leakage and keeping each representation clean and disentangled. Moreover, to
model the conditional distribution over future actions, we employ a
diffusion-based transformer that disentangles action representations from
shared latent features. Extensive experiments on both real-world and simulation
environments demonstrate that DreamVLA achieves 76.7% success rate on real
robot tasks and 4.44 average length on the CALVIN ABC-D benchmarks.
686cab67364e2ad167eb5464
Wenyao Zhang
admin_assigned
2025-07-08T13:54:59.906Z
65f9533b136fb8ddbd14e1fa
/avatars/d88f75da0448093ccd1babba2a37d73f.svg
Zhang
user
WenyaoZhang
686cab67364e2ad167eb5465
Hongsi Liu
claimed_verified
2025-07-09T08:51:30.389Z
6672e6541af6127e6b0e9edb
/avatars/f401cf6f8b55276788c162d9ed25886f.svg
EricLiu
user
EricLiuhhh
686cab67364e2ad167eb5466
Zekun Qi
claimed_verified
2025-07-08T08:05:57.749Z
63c3e8abc7d7f4c63a515a02
https://cdn-avatars.huggingface.co/v1/production/uploads/63c3e8abc7d7f4c63a515a02/npMHnVP2hHLbvoUGe7C4O.jpeg
Zekun Qi
user
qizekun
686cab67364e2ad167eb5467
Yunnan Wang
admin_assigned
2025-07-08T13:55:17.010Z
674d7c11c3df11ec9a1f3c17
/avatars/daa392c220dfc1ea791b9d48385df2f6.svg
Yunnan Wang
user
wangyunnan
686cab67364e2ad167eb5468
XinQiang Yu
686cab67364e2ad167eb5469
Jiazhao Zhang
admin_assigned
2025-07-08T13:55:28.159Z
65658233d35fc55406e8b00d
/avatars/660eaa1923cd3e3478cec8197936a75c.svg
Jiazhao Zhang
user
Jzzhang
686cab67364e2ad167eb546a
Runpei Dong
claimed_verified
2025-07-08T08:06:00.401Z
6201fc5d91d53938a6432fbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6201fc5d91d53938a6432fbf/VLs8ZYaZrop4KBpZn53fH.jpeg
Runpei Dong
user
RunpeiDong
686cab67364e2ad167eb546b
Jiawei He
686cab67364e2ad167eb546c
He Wang
686cab67364e2ad167eb546d
Zhizheng Zhang
686cab67364e2ad167eb546e
Li Yi
686cab67364e2ad167eb546f
Wenjun Zeng
686cab67364e2ad167eb5470
Xin Jin
DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive
  World Knowledge
Recent advances in vision-language-action (VLA) models have shown promise in
integrating image generation with action prediction to improve generalization
and reasoning in robot manipulation. However, existing methods are limited to
challenging image-based forecasting, which suffers from redundant information
and lacks comprehensive and critical world knowledge, including dynamic,
spatial and semantic information. To address these limitations, we propose
DreamVLA, a novel VLA framework that integrates comprehensive world knowledge
forecasting to enable inverse dynamics modeling, thereby establishing a
perception-prediction-action loop for manipulation tasks. Specifically,
DreamVLA introduces a dynamic-region-guided world knowledge prediction,
integrated with the spatial and semantic cues, which provide compact yet
comprehensive representations for action planning. This design aligns with how
humans interact with the world by first forming abstract multimodal reasoning
chains before acting. To mitigate interference among the dynamic, spatial and
semantic information during training, we adopt a block-wise structured
attention mechanism that masks their mutual attention, preventing information
leakage and keeping each representation clean and disentangled. Moreover, to
model the conditional distribution over future actions, we employ a
diffusion-based transformer that disentangles action representations from
shared latent features. Extensive experiments on both real-world and simulation
environments demonstrate that DreamVLA achieves 76.7% success rate on real
robot tasks and 4.44 average length on the CALVIN ABC-D benchmarks.
DreamVLA integrates comprehensive world knowledge forecasting with a block-wise structured attention mechanism and diffusion-based transformer to improve action prediction and generalization in robot manipulation tasks.
vision-language-action (VLA) models image generation action prediction inverse dynamics modeling dynamic-region-guided world knowledge prediction spatial and semantic cues block-wise structured attention mechanism diffusion-based transformer CALVIN ABC-D benchmarks