Compositional Video Generation as Flow Equalization
Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated
unprecedented capability to transform natural language descriptions into
stunning and photorealistic videos. Despite the promising results, a
significant challenge remains: these models struggle to fully grasp complex
compositional interactions between multiple concepts and actions. This issue
arises when some words dominantly influence the final video, overshadowing
other concepts.To tackle this problem, we introduce Vico, a generic
framework for compositional video generation that explicitly ensures all
concepts are represented properly. At its core, Vico analyzes how input tokens
influence the generated video, and adjusts the model to prevent any single
concept from dominating. Specifically, Vico extracts attention weights from all
layers to build a spatial-temporal attention graph, and then estimates the
influence as the max-flow from the source text token to the video target
token. Although the direct computation of attention flow in diffusion models is
typically infeasible, we devise an efficient approximation based on subgraph
flows and employ a fast and vectorized implementation, which in turn makes the
flow computation manageable and differentiable. By updating the noisy latent to
balance these flows, Vico captures complex interactions and consequently
produces videos that closely adhere to textual descriptions. We apply our
method to multiple diffusion-based video models for compositional T2V and video
editing. Empirical results demonstrate that our framework significantly
enhances the compositional richness and accuracy of the generated videos. Visit
our website
at~https://adamdad.github.io/vico/{https://adamdad.github.io/vico/}.
668c9d937dbe0ab21d9a65f3
Xingyi Yang
admin_assigned
2024-07-09T21:07:35.186Z
634cfebc350bcee9bed20a4d
https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
Xingyi Yang
user
adamdad
668c9d937dbe0ab21d9a65f4
Xinchao Wang
admin_assigned
2024-07-09T21:08:16.858Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
Compositional Video Generation as Flow Equalization
Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated
unprecedented capability to transform natural language descriptions into
stunning and photorealistic videos. Despite the promising results, a
significant challenge remains: these models struggle to fully grasp complex
compositional interactions between multiple concepts and actions. This issue
arises when some words dominantly influence the final video, overshadowing
other concepts.To tackle this problem, we introduce Vico, a generic
framework for compositional video generation that explicitly ensures all
concepts are represented properly. At its core, Vico analyzes how input tokens
influence the generated video, and adjusts the model to prevent any single
concept from dominating. Specifically, Vico extracts attention weights from all
layers to build a spatial-temporal attention graph, and then estimates the
influence as the max-flow from the source text token to the video target
token. Although the direct computation of attention flow in diffusion models is
typically infeasible, we devise an efficient approximation based on subgraph
flows and employ a fast and vectorized implementation, which in turn makes the
flow computation manageable and differentiable. By updating the noisy latent to
balance these flows, Vico captures complex interactions and consequently
produces videos that closely adhere to textual descriptions. We apply our
method to multiple diffusion-based video models for compositional T2V and video
editing. Empirical results demonstrate that our framework significantly
enhances the compositional richness and accuracy of the generated videos. Visit
our website
at~https://adamdad.github.io/vico/{https://adamdad.github.io/vico/}.
Vico is a framework that enhances text-to-video diffusion models by ensuring balanced representation of all concepts in the generated video through spatial-temporal attention graphs and max-flow approximations.
diffusion models compositional video generation input tokens attention weights spatial-temporal attention graph max-flow subgraph flows vectorized implementation noisy latent compositional T2V video editing