GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via
  Blender-Oriented GPT Planning
Recent advances in text-to-video generation have harnessed the power of
diffusion models to create visually compelling content conditioned on text
prompts. However, they usually encounter high computational costs and often
struggle to produce videos with coherent physical motions. To tackle these
issues, we propose GPT4Motion, a training-free framework that leverages the
planning capability of large language models such as GPT, the physical
simulation strength of Blender, and the excellent image generation ability of
text-to-image diffusion models to enhance the quality of video synthesis.
Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a
user textual prompt, which commands Blender's built-in physics engine to craft
fundamental scene components that encapsulate coherent physical motions across
frames. Then these components are inputted into Stable Diffusion to generate a
video aligned with the textual prompt. Experimental results on three basic
physical motion scenarios, including rigid object drop and collision, cloth
draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate
high-quality videos efficiently in maintaining motion coherency and entity
consistency. GPT4Motion offers new insights in text-to-video research,
enhancing its quality and broadening its horizon for future explorations.
655d7a5e16ede91c780ce244
Jiaxi Lv
claimed_verified
2023-12-07T08:30:15.836Z
63c15f0ef02ef5b95e102d1e
/avatars/ea50adfdb91de8545dbbc3e06855a468.svg
JaX lyu
user
jax2000
655d7a5e16ede91c780ce245
Yi Huang
655d7a5e16ede91c780ce246
Mingfu Yan
admin_assigned
2023-11-22T13:44:28.238Z
64fd99df66a284440bcdb46d
/avatars/aa0d0556385d14ad4b31d7ecea20643f.svg
Mingfu Yan
user
MingfuYAN
655d7a5e16ede91c780ce247
Jiancheng Huang
admin_assigned
2023-11-22T13:44:37.374Z
6395e7c311074713ad1fc29e
/avatars/6b1e61aaaed89af05daaf15f90747976.svg
jiancheng huang
user
jiancheng
655d7a5e16ede91c780ce248
Jianzhuang Liu
655d7a5e16ede91c780ce249
Yifan Liu
655d7a5e16ede91c780ce24a
Yafei Wen
655d7a5e16ede91c780ce24b
Xiaoxin Chen
admin_assigned
2023-11-22T13:55:11.798Z
65389a669c474315d7425f96
/avatars/2fa3828ca489cfe1948129a0eccf264f.svg
chenxiaoxin
user
steelozazala
655d7a5e16ede91c780ce24c
Shifeng Chen
admin_assigned
2023-11-22T13:55:23.457Z
6393446923aec2f4ee593925
/avatars/6570a7ce6f6495ba47eda51e7c70c8ce.svg
chenshifeng
user
suifeng
GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via
  Blender-Oriented GPT Planning
Recent advances in text-to-video generation have harnessed the power of
diffusion models to create visually compelling content conditioned on text
prompts. However, they usually encounter high computational costs and often
struggle to produce videos with coherent physical motions. To tackle these
issues, we propose GPT4Motion, a training-free framework that leverages the
planning capability of large language models such as GPT, the physical
simulation strength of Blender, and the excellent image generation ability of
text-to-image diffusion models to enhance the quality of video synthesis.
Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a
user textual prompt, which commands Blender's built-in physics engine to craft
fundamental scene components that encapsulate coherent physical motions across
frames. Then these components are inputted into Stable Diffusion to generate a
video aligned with the textual prompt. Experimental results on three basic
physical motion scenarios, including rigid object drop and collision, cloth
draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate
high-quality videos efficiently in maintaining motion coherency and entity
consistency. GPT4Motion offers new insights in text-to-video research,
enhancing its quality and broadening its horizon for future explorations.
GPT4Motion integrates GPT, Blender, and Stable Diffusion to generate high-quality, motion-coherent videos from text prompts more efficiently than previous methods.
diffusion models GPT Blender physics engine Stable Diffusion rigid object drop and collision cloth draping and swinging liquid flow