Dita: Scaling Diffusion Transformer for Generalist
  Vision-Language-Action Policy
While recent vision-language-action models trained on diverse robot datasets
exhibit promising generalization capabilities with limited in-domain data,
their reliance on compact action heads to predict discretized or continuous
actions constrains adaptability to heterogeneous action spaces. We present
Dita, a scalable framework that leverages Transformer architectures to directly
denoise continuous action sequences through a unified multimodal diffusion
process. Departing from prior methods that condition denoising on fused
embeddings via shallow networks, Dita employs in-context conditioning --
enabling fine-grained alignment between denoised actions and raw visual tokens
from historical observations. This design explicitly models action deltas and
environmental nuances. By scaling the diffusion action denoiser alongside the
Transformer's scalability, Dita effectively integrates cross-embodiment
datasets across diverse camera perspectives, observation scenes, tasks, and
action spaces. Such synergy enhances robustness against various variances and
facilitates the successful execution of long-horizon tasks. Evaluations across
extensive benchmarks demonstrate state-of-the-art or comparative performance in
simulation. Notably, Dita achieves robust real-world adaptation to
environmental variances and complex long-horizon tasks through 10-shot
finetuning, using only third-person camera inputs. The architecture establishes
a versatile, lightweight and open-source baseline for generalist robot policy
learning. Project Page: https://robodita.github.io.
67e3e1e20706b07bfb2713d6
Zhi Hou
admin_assigned
2025-03-27T09:43:49.995Z
643fa1c318afbc4d1f3e5e59
/avatars/f8f35355902b4cda72e9c6d768322fae.svg
Zhi Hou
user
zhihou
67e3e1e20706b07bfb2713d7
Tianyi Zhang
admin_assigned
2025-03-27T09:44:21.118Z
64c9e86a6a26cddbecd9bae2
/avatars/61a84989dbbc1898ebcba3236dbed039.svg
Tianyi Zhang
user
TianyiZhang0213
67e3e1e20706b07bfb2713d8
Yuwen Xiong
67e3e1e20706b07bfb2713d9
Haonan Duan
claimed_verified
2025-03-27T09:04:53.498Z
66ab30dfd456f0408b93f27b
https://cdn-avatars.huggingface.co/v1/production/uploads/66ab30dfd456f0408b93f27b/nps4Kni_eOExO5Z92RiiF.jpeg
Haonan Duan
user
robot-haonan
67e3e1e20706b07bfb2713da
Hengjun Pu
admin_assigned
2025-03-27T09:45:01.725Z
648a1e44fe11ebd7489c289c
https://cdn-avatars.huggingface.co/v1/production/uploads/648a1e44fe11ebd7489c289c/WvszTywGnuY8OV4500_qh.jpeg
Hengjun Pu
user
MIASANMIA
67e3e1e20706b07bfb2713db
Ronglei Tong
admin_assigned
2025-03-27T09:45:07.673Z
66b9a5bb32be421cd8538cd6
/avatars/a1f7c0fe3ed4741017db713b4e6d47c8.svg
Ronglei Tong
user
TTTTTony
67e3e1e20706b07bfb2713dc
Chengyang Zhao
admin_assigned
2025-03-27T09:45:13.906Z
679165b9c7f527ef3619504e
/avatars/f3e6ce5fc3d05c8632d8b208f55c2987.svg
Chengyang Zhao
user
chengyzhao
67e3e1e20706b07bfb2713dd
Xizhou Zhu
admin_assigned
2025-03-27T09:45:20.614Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
67e3e1e20706b07bfb2713de
Yu Qiao
67e3e1e20706b07bfb2713df
Jifeng Dai
admin_assigned
2025-03-27T09:45:27.039Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
67e3e1e20706b07bfb2713e0
Yuntao Chen
admin_assigned
2025-03-27T09:45:40.200Z
632dab84fdb35759ea6646a0
/avatars/857b0b4d115aa5ab2f143e60b0e4edc6.svg
Yuntao Chen
user
YuntaoChen
Dita: Scaling Diffusion Transformer for Generalist
  Vision-Language-Action Policy
While recent vision-language-action models trained on diverse robot datasets
exhibit promising generalization capabilities with limited in-domain data,
their reliance on compact action heads to predict discretized or continuous
actions constrains adaptability to heterogeneous action spaces. We present
Dita, a scalable framework that leverages Transformer architectures to directly
denoise continuous action sequences through a unified multimodal diffusion
process. Departing from prior methods that condition denoising on fused
embeddings via shallow networks, Dita employs in-context conditioning --
enabling fine-grained alignment between denoised actions and raw visual tokens
from historical observations. This design explicitly models action deltas and
environmental nuances. By scaling the diffusion action denoiser alongside the
Transformer's scalability, Dita effectively integrates cross-embodiment
datasets across diverse camera perspectives, observation scenes, tasks, and
action spaces. Such synergy enhances robustness against various variances and
facilitates the successful execution of long-horizon tasks. Evaluations across
extensive benchmarks demonstrate state-of-the-art or comparative performance in
simulation. Notably, Dita achieves robust real-world adaptation to
environmental variances and complex long-horizon tasks through 10-shot
finetuning, using only third-person camera inputs. The architecture establishes
a versatile, lightweight and open-source baseline for generalist robot policy
learning. Project Page: https://robodita.github.io.
Dita, a scalable framework using Transformer architectures and multimodal diffusion, achieves state-of-the-art performance with fine-grained action denoising and real-world adaptation.
Transformer architectures multimodal diffusion denoising in-context conditioning action deltas cross-embodiment datasets long-horizon tasks 10-shot finetuning