DreamRunner: Fine-Grained Storytelling Video Generation with
  Retrieval-Augmented Motion Adaptation
Storytelling video generation (SVG) has recently emerged as a task to create
long, multi-motion, multi-scene videos that consistently represent the story
described in the input text script. SVG holds great potential for diverse
content creation in media and entertainment; however, it also presents
significant challenges: (1) objects must exhibit a range of fine-grained,
complex motions, (2) multiple objects need to appear consistently across
scenes, and (3) subjects may require multiple motions with seamless transitions
within a single scene. To address these challenges, we propose DreamRunner, a
novel story-to-video generation method: First, we structure the input script
using a large language model (LLM) to facilitate both coarse-grained scene
planning as well as fine-grained object-level layout and motion planning. Next,
DreamRunner presents retrieval-augmented test-time adaptation to capture target
motion priors for objects in each scene, supporting diverse motion
customization based on retrieved videos, thus facilitating the generation of
new videos with complex, scripted motions. Lastly, we propose a novel
spatial-temporal region-based 3D attention and prior injection module SR3AI for
fine-grained object-motion binding and frame-by-frame semantic control. We
compare DreamRunner with various SVG baselines, demonstrating state-of-the-art
performance in character consistency, text alignment, and smooth transitions.
Additionally, DreamRunner exhibits strong fine-grained condition-following
ability in compositional text-to-video generation, significantly outperforming
baselines on T2V-ComBench. Finally, we validate DreamRunner's robust ability to
generate multi-object interactions with qualitative examples.
674539fa5cb54d8d661272cc
Zun Wang
admin_assigned
2024-11-26T22:43:26.509Z
6622c710b0e5c5e3de8311c1
/avatars/a824c150040731679bbd77762ca9d4eb.svg
Zun Wang
user
ZunWang
674539fa5cb54d8d661272cd
Jialu Li
claimed_verified
2024-11-26T21:21:11.019Z
65d69eb0bcf1fed72325a052
/avatars/5ef735c3137555bb52e512ae820e437c.svg
Jialu Li
user
jialuliluka
674539fa5cb54d8d661272ce
Han Lin
claimed_verified
2024-12-02T08:51:14.248Z
646e86350867c99c2d3f2ecf
/avatars/b89798ff623abffb169eacda2ac32fde.svg
Han Lin
user
hanlincs
674539fa5cb54d8d661272cf
Jaehong Yoon
claimed_verified
2024-11-26T21:21:09.391Z
652066649004117947e46ed6
/avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
Jaehong Yoon
user
jaehong31
674539fa5cb54d8d661272d0
Mohit Bansal
admin_assigned
2024-11-26T22:43:42.862Z
665d9d3a057f7c508f98c625
https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
Mohit Bansal
user
mohitbansal
DreamRunner: Fine-Grained Storytelling Video Generation with
  Retrieval-Augmented Motion Adaptation
Storytelling video generation (SVG) has recently emerged as a task to create
long, multi-motion, multi-scene videos that consistently represent the story
described in the input text script. SVG holds great potential for diverse
content creation in media and entertainment; however, it also presents
significant challenges: (1) objects must exhibit a range of fine-grained,
complex motions, (2) multiple objects need to appear consistently across
scenes, and (3) subjects may require multiple motions with seamless transitions
within a single scene. To address these challenges, we propose DreamRunner, a
novel story-to-video generation method: First, we structure the input script
using a large language model (LLM) to facilitate both coarse-grained scene
planning as well as fine-grained object-level layout and motion planning. Next,
DreamRunner presents retrieval-augmented test-time adaptation to capture target
motion priors for objects in each scene, supporting diverse motion
customization based on retrieved videos, thus facilitating the generation of
new videos with complex, scripted motions. Lastly, we propose a novel
spatial-temporal region-based 3D attention and prior injection module SR3AI for
fine-grained object-motion binding and frame-by-frame semantic control. We
compare DreamRunner with various SVG baselines, demonstrating state-of-the-art
performance in character consistency, text alignment, and smooth transitions.
Additionally, DreamRunner exhibits strong fine-grained condition-following
ability in compositional text-to-video generation, significantly outperforming
baselines on T2V-ComBench. Finally, we validate DreamRunner's robust ability to
generate multi-object interactions with qualitative examples.
DreamRunner integrates scene structuring, motion retrieval, and spatial-temporal attention to generate coherent, motion-rich storytelling videos from text scripts.
storytelling video generation SVG large language model scene planning object-level layout motion planning retrieval-augmented test-time adaptation target motion priors spatial-temporal region-based 3D attention prior injection module SR3AI character consistency text alignment smooth transitions compositional text-to-video generation T2V-ComBench multi-object interactions