TC4D: Trajectory-Conditioned Text-to-4D Generation
Recent techniques for text-to-4D generation synthesize dynamic 3D scenes
using supervision from pre-trained text-to-video models. However, existing
representations for motion, such as deformation models or time-dependent neural
representations, are limited in the amount of motion they can generate-they
cannot synthesize motion extending far beyond the bounding box used for volume
rendering. The lack of a more flexible motion model contributes to the gap in
realism between 4D generation methods and recent, near-photorealistic video
generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D
generation, which factors motion into global and local components. We represent
the global motion of a scene's bounding box using rigid transformation along a
trajectory parameterized by a spline. We learn local deformations that conform
to the global trajectory using supervision from a text-to-video model. Our
approach enables the synthesis of scenes animated along arbitrary trajectories,
compositional scene generation, and significant improvements to the realism and
amount of generated motion, which we evaluate qualitatively and through a user
study. Video results can be viewed on our website:
https://sherwinbahmani.github.io/tc4d.
6603909490c62bb38f073051
Sherwin Bahmani
admin_assigned
2024-03-27T09:47:20.297Z
647ef66c45baf21ad707b291
/avatars/4cd941cdca6dd829fdc9cb3fb788a99c.svg
Sherwin Bahmani
user
sherwinbahmani
6603909490c62bb38f073052
Xian Liu
admin_assigned
2024-03-27T09:47:36.756Z
6423f10a3fa01ecba6fca861
/avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
Xian Liu
user
alvinliu0
6603909490c62bb38f073053
Yifan Wang
6603909490c62bb38f073054
Ivan Skorokhodov
admin_assigned
2024-03-27T09:48:10.431Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
6603909490c62bb38f073055
Victor Rong
6603909490c62bb38f073056
Ziwei Liu
admin_assigned
2024-03-27T09:48:20.683Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
6603909490c62bb38f073057
Xihui Liu
admin_assigned
2024-03-27T09:48:26.669Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
6603909490c62bb38f073058
Jeong Joon Park
claimed_verified
2024-07-11T17:34:50.085Z
6517d7f84fadbeb643c68269
/avatars/ad21288123abcd4488ae1bdcb10b54e7.svg
JJ Park
user
jjparkcv
6603909490c62bb38f073059
Sergey Tulyakov
6603909490c62bb38f07305a
Gordon Wetzstein
6603909490c62bb38f07305b
Andrea Tagliasacchi
6603909490c62bb38f07305c
David B. Lindell
TC4D: Trajectory-Conditioned Text-to-4D Generation
Recent techniques for text-to-4D generation synthesize dynamic 3D scenes
using supervision from pre-trained text-to-video models. However, existing
representations for motion, such as deformation models or time-dependent neural
representations, are limited in the amount of motion they can generate-they
cannot synthesize motion extending far beyond the bounding box used for volume
rendering. The lack of a more flexible motion model contributes to the gap in
realism between 4D generation methods and recent, near-photorealistic video
generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D
generation, which factors motion into global and local components. We represent
the global motion of a scene's bounding box using rigid transformation along a
trajectory parameterized by a spline. We learn local deformations that conform
to the global trajectory using supervision from a text-to-video model. Our
approach enables the synthesis of scenes animated along arbitrary trajectories,
compositional scene generation, and significant improvements to the realism and
amount of generated motion, which we evaluate qualitatively and through a user
study. Video results can be viewed on our website:
https://sherwinbahmani.github.io/tc4d.
TC4D, a trajectory-conditioned text-to-4D generation method, improves motion realism and synthesis by separating global motion into rigid transformations and learning local deformations using text-to-video supervision.
trajectory-conditioned text-to-4D generation rigid transformation spline local deformations compositional scene generation user study