3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video
  Generation
This paper aims to manipulate multi-entity 3D motions in video generation.
Previous methods on controllable video generation primarily leverage 2D control
signals to manipulate object motions and have achieved remarkable synthesis
results. However, 2D control signals are inherently limited in expressing the
3D nature of object motions. To overcome this problem, we introduce
3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D
space, given user-desired 6DoF pose (location and rotation) sequences of
entities. At the core of our approach is a plug-and-play 3D-motion grounded
object injector that fuses multiple input entities with their respective 3D
trajectories through a gated self-attention mechanism. In addition, we exploit
an injector architecture to preserve the video diffusion prior, which is
crucial for generalization ability. To mitigate video quality degradation, we
introduce a domain adaptor during training and employ an annealed sampling
strategy during inference. To address the lack of suitable training data, we
construct a 360-Motion Dataset, which first correlates collected 3D human and
animal assets with GPT-generated trajectory and then captures their motion with
12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments
show that 3DTrajMaster sets a new state-of-the-art in both accuracy and
generalization for controlling multi-entity 3D motions. Project page:
http://fuxiao0719.github.io/projects/3dtrajmaster
675962a407803203aef900ec
Xiao Fu
claimed_verified
2024-12-11T10:09:04.015Z
63aef2cafcca84593e6682db
https://cdn-avatars.huggingface.co/v1/production/uploads/1672409763337-noauth.jpeg
Xiao Fu
user
lemonaddie
675962a407803203aef900ed
Xian Liu
675962a407803203aef900ee
Xintao Wang
admin_assigned
2024-12-11T13:21:32.060Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
675962a407803203aef900ef
Sida Peng
admin_assigned
2024-12-11T13:21:24.280Z
62986ca2b58e71e2ac9b8f01
/avatars/83944db5f3dbb6f47c47c46fb2cb2849.svg
Sida Peng
user
pengsida
675962a407803203aef900f0
Menghan Xia
admin_assigned
2024-12-11T13:20:48.288Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
675962a407803203aef900f1
Xiaoyu Shi
admin_assigned
2024-12-11T13:21:17.084Z
641af5fcf902cc42730b47e2
/avatars/73ac99dec226f0e814a16d2f1dbfbce8.svg
Xiaoyu Shi
user
btwbtm
675962a407803203aef900f2
Ziyang Yuan
admin_assigned
2024-12-11T13:20:31.083Z
66c15837b186e4f6a0dac80c
/avatars/a3b75d6945f1608e64a2fcff887a5024.svg
Ziyang Yuan
user
ziyangy
675962a407803203aef900f3
Pengfei Wan
claimed_verified
2025-03-17T14:43:06.988Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
675962a407803203aef900f4
Di Zhang
admin_assigned
2024-12-11T13:20:13.824Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
675962a407803203aef900f5
Dahua Lin
admin_assigned
2024-12-11T13:19:40.425Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video
  Generation
This paper aims to manipulate multi-entity 3D motions in video generation.
Previous methods on controllable video generation primarily leverage 2D control
signals to manipulate object motions and have achieved remarkable synthesis
results. However, 2D control signals are inherently limited in expressing the
3D nature of object motions. To overcome this problem, we introduce
3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D
space, given user-desired 6DoF pose (location and rotation) sequences of
entities. At the core of our approach is a plug-and-play 3D-motion grounded
object injector that fuses multiple input entities with their respective 3D
trajectories through a gated self-attention mechanism. In addition, we exploit
an injector architecture to preserve the video diffusion prior, which is
crucial for generalization ability. To mitigate video quality degradation, we
introduce a domain adaptor during training and employ an annealed sampling
strategy during inference. To address the lack of suitable training data, we
construct a 360-Motion Dataset, which first correlates collected 3D human and
animal assets with GPT-generated trajectory and then captures their motion with
12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments
show that 3DTrajMaster sets a new state-of-the-art in both accuracy and
generalization for controlling multi-entity 3D motions. Project page:
http://fuxiao0719.github.io/projects/3dtrajmaster
3DTrajMaster controls multi-entity 3D motions in video generation using a 3D-motion grounded object injector with gated self-attention, while maintaining video diffusion priors and mitigating quality degradation.
3DTrajMaster 3D-motion grounded object injector gated self-attention video diffusion prior annealed sampling strategy 360-Motion Dataset GPT-generated trajectory 3D UE platforms