Learning Video Generation for Robotic Manipulation with Collaborative
  Trajectory Control
Recent advances in video diffusion models have demonstrated strong potential
for generating robotic decision-making data, with trajectory conditions further
enabling fine-grained control. However, existing trajectory-based methods
primarily focus on individual object motion and struggle to capture
multi-object interaction crucial in complex robotic manipulation. This
limitation arises from multi-feature entanglement in overlapping regions, which
leads to degraded visual fidelity. To address this, we present RoboMaster, a
novel framework that models inter-object dynamics through a collaborative
trajectory formulation. Unlike prior methods that decompose objects, our core
is to decompose the interaction process into three sub-stages: pre-interaction,
interaction, and post-interaction. Each stage is modeled using the feature of
the dominant object, specifically the robotic arm in the pre- and
post-interaction phases and the manipulated object during interaction, thereby
mitigating the drawback of multi-object feature fusion present during
interaction in prior work. To further ensure subject semantic consistency
throughout the video, we incorporate appearance- and shape-aware latent
representations for objects. Extensive experiments on the challenging Bridge V2
dataset, as well as in-the-wild evaluation, demonstrate that our method
outperforms existing approaches, establishing new state-of-the-art performance
in trajectory-controlled video generation for robotic manipulation.
683e6b6424742a21489ec9f8
Xiao Fu
683e6b6424742a21489ec9f9
Xintao Wang
admin_assigned
2025-06-03T12:36:21.590Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
683e6b6424742a21489ec9fa
Xian Liu
683e6b6424742a21489ec9fb
Jianhong Bai
admin_assigned
2025-06-03T12:36:29.349Z
6530bf50f145530101ec03a2
/avatars/c61c00c314cf202b64968e51e855694d.svg
Jianhong Bai
user
jianhongbai
683e6b6424742a21489ec9fc
Runsen Xu
admin_assigned
2025-06-03T12:36:13.754Z
6458b1103b81018d6b93defb
https://cdn-avatars.huggingface.co/v1/production/uploads/6458b1103b81018d6b93defb/JFr5j-3UO0xxlEdUksikU.jpeg
Runsen Xu
user
RunsenXu
683e6b6424742a21489ec9fd
Pengfei Wan
683e6b6424742a21489ec9fe
Di Zhang
683e6b6424742a21489ec9ff
Dahua Lin
admin_assigned
2025-06-03T12:35:57.663Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
Learning Video Generation for Robotic Manipulation with Collaborative
  Trajectory Control
Recent advances in video diffusion models have demonstrated strong potential
for generating robotic decision-making data, with trajectory conditions further
enabling fine-grained control. However, existing trajectory-based methods
primarily focus on individual object motion and struggle to capture
multi-object interaction crucial in complex robotic manipulation. This
limitation arises from multi-feature entanglement in overlapping regions, which
leads to degraded visual fidelity. To address this, we present RoboMaster, a
novel framework that models inter-object dynamics through a collaborative
trajectory formulation. Unlike prior methods that decompose objects, our core
is to decompose the interaction process into three sub-stages: pre-interaction,
interaction, and post-interaction. Each stage is modeled using the feature of
the dominant object, specifically the robotic arm in the pre- and
post-interaction phases and the manipulated object during interaction, thereby
mitigating the drawback of multi-object feature fusion present during
interaction in prior work. To further ensure subject semantic consistency
throughout the video, we incorporate appearance- and shape-aware latent
representations for objects. Extensive experiments on the challenging Bridge V2
dataset, as well as in-the-wild evaluation, demonstrate that our method
outperforms existing approaches, establishing new state-of-the-art performance
in trajectory-controlled video generation for robotic manipulation.
A novel framework, RoboMaster, enhances trajectory-controlled video generation for robotic manipulation by modeling inter-object dynamics through a collaborative trajectory formulation, achieving state-of-the-art performance on the Bridge V2 dataset.
video diffusion models trajectory conditions multi-object interaction multi-feature entanglement visual fidelity collaborative trajectory formulation pre-interaction interaction post-interaction appearance-aware latent representations shape-aware latent representations trajectory-controlled video generation robotic manipulation Bridge V2 dataset