GenMAC: Compositional Text-to-Video Generation with Multi-Agent
  Collaboration
Text-to-video generation models have shown significant progress in the recent
years. However, they still struggle with generating complex dynamic scenes
based on compositional text prompts, such as attribute binding for multiple
objects, temporal dynamics associated with different objects, and interactions
between objects. Our key motivation is that complex tasks can be decomposed
into simpler ones, each handled by a role-specialized MLLM agent. Multiple
agents can collaborate together to achieve collective intelligence for complex
goals. We propose GenMAC, an iterative, multi-agent framework that enables
compositional text-to-video generation. The collaborative workflow includes
three stages: Design, Generation, and Redesign, with an iterative loop between
the Generation and Redesign stages to progressively verify and refine the
generated videos. The Redesign stage is the most challenging stage that aims to
verify the generated videos, suggest corrections, and redesign the text
prompts, frame-wise layouts, and guidance scales for the next iteration of
generation. To avoid hallucination of a single MLLM agent, we decompose this
stage to four sequentially-executed MLLM-based agents: verification agent,
suggestion agent, correction agent, and output structuring agent. Furthermore,
to tackle diverse scenarios of compositional text-to-video generation, we
design a self-routing mechanism to adaptively select the proper correction
agent from a collection of correction agents each specialized for one scenario.
Extensive experiments demonstrate the effectiveness of GenMAC, achieving
state-of-the art performance in compositional text-to-video generation.
6752a1062f16c25ac96879dc
Kaiyi Huang
admin_assigned
2024-12-06T17:48:59.205Z
637cba13b8e573d75be96ea6
/avatars/5eca230e63d66947b2a05c1ff964a96c.svg
Nina
user
NinaKarine
6752a1062f16c25ac96879dd
Yukun Huang
claimed_verified
2024-12-12T14:34:31.663Z
638ee900ee7e45e0474a5712
https://cdn-avatars.huggingface.co/v1/production/uploads/638ee900ee7e45e0474a5712/KLli_eCbWwffKR7oLDmV3.jpeg
Yukun Huang
user
KevinHuang
6752a1062f16c25ac96879de
Xuefei Ning
claimed_verified
2024-12-30T19:37:18.692Z
641031b1a78453b8d96b8420
https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
Xuefei Ning
user
Foxfi
6752a1062f16c25ac96879df
Zinan Lin
claimed_verified
2024-12-09T11:07:13.931Z
64c832a8c547ed5243d29630
https://cdn-avatars.huggingface.co/v1/production/uploads/64c832a8c547ed5243d29630/a46V0xOVRVknM9mUjKkTf.jpeg
Zinan Lin
user
fjxmlzn
6752a1062f16c25ac96879e0
Yu Wang
6752a1062f16c25ac96879e1
Xihui Liu
claimed_verified
2025-05-26T08:44:05.510Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
GenMAC: Compositional Text-to-Video Generation with Multi-Agent
  Collaboration
Text-to-video generation models have shown significant progress in the recent
years. However, they still struggle with generating complex dynamic scenes
based on compositional text prompts, such as attribute binding for multiple
objects, temporal dynamics associated with different objects, and interactions
between objects. Our key motivation is that complex tasks can be decomposed
into simpler ones, each handled by a role-specialized MLLM agent. Multiple
agents can collaborate together to achieve collective intelligence for complex
goals. We propose GenMAC, an iterative, multi-agent framework that enables
compositional text-to-video generation. The collaborative workflow includes
three stages: Design, Generation, and Redesign, with an iterative loop between
the Generation and Redesign stages to progressively verify and refine the
generated videos. The Redesign stage is the most challenging stage that aims to
verify the generated videos, suggest corrections, and redesign the text
prompts, frame-wise layouts, and guidance scales for the next iteration of
generation. To avoid hallucination of a single MLLM agent, we decompose this
stage to four sequentially-executed MLLM-based agents: verification agent,
suggestion agent, correction agent, and output structuring agent. Furthermore,
to tackle diverse scenarios of compositional text-to-video generation, we
design a self-routing mechanism to adaptively select the proper correction
agent from a collection of correction agents each specialized for one scenario.
Extensive experiments demonstrate the effectiveness of GenMAC, achieving
state-of-the art performance in compositional text-to-video generation.
GenMAC, a multi-agent framework, enhances text-to-video generation by decomposing complex tasks into simpler stages managed by specialized agents, ensuring high-quality and accurate output through iterative refinement.
text-to-video generation MLLM compositional text prompts iterative framework collaborative workflow verification agent suggestion agent correction agent output structuring agent self-routing mechanism