Anim-Director: A Large Multimodal Model Powered Agent for Controllable
  Animation Video Generation
Traditional animation generation methods depend on training generative models
with human-labelled data, entailing a sophisticated multi-stage pipeline that
demands substantial human effort and incurs high training costs. Due to limited
prompting plans, these methods typically produce brief, information-poor, and
context-incoherent animations. To overcome these limitations and automate the
animation process, we pioneer the introduction of large multimodal models
(LMMs) as the core processor to build an autonomous animation-making agent,
named Anim-Director. This agent mainly harnesses the advanced understanding and
reasoning capabilities of LMMs and generative AI tools to create animated
videos from concise narratives or simple instructions. Specifically, it
operates in three main stages: Firstly, the Anim-Director generates a coherent
storyline from user inputs, followed by a detailed director's script that
encompasses settings of character profiles and interior/exterior descriptions,
and context-coherent scene descriptions that include appearing characters,
interiors or exteriors, and scene events. Secondly, we employ LMMs with the
image generation tool to produce visual images of settings and scenes. These
images are designed to maintain visual consistency across different scenes
using a visual-language prompting method that combines scene descriptions and
images of the appearing character and setting. Thirdly, scene images serve as
the foundation for producing animated videos, with LMMs generating prompts to
guide this process. The whole process is notably autonomous without manual
intervention, as the LMMs interact seamlessly with generative tools to generate
prompts, evaluate visual quality, and select the best one to optimize the final
output.
66c84b3a380e8fb44acee64d
Yunxin Li
extracted_pending
2024-08-23T08:41:33.316Z
62fdb01bc1588e1d4c6c1a7c
/avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
Yunxin Li
user
YunxinLi
66c84b3a380e8fb44acee64e
Haoyuan Shi
claimed_verified
2025-06-13T07:44:00.602Z
652fb8bcc9dd2692a25ef2e3
/avatars/461e6cc1c3441cde18192b080b0b8576.svg
Haoyuan Shi
user
MrSunshy
66c84b3a380e8fb44acee64f
Baotian Hu
66c84b3a380e8fb44acee650
Longyue Wang
66c84b3a380e8fb44acee651
Jiashun Zhu
66c84b3a380e8fb44acee652
Jinyi Xu
66c84b3a380e8fb44acee653
Zhen Zhao
66c84b3a380e8fb44acee654
Min Zhang
Anim-Director: A Large Multimodal Model Powered Agent for Controllable
  Animation Video Generation
Traditional animation generation methods depend on training generative models
with human-labelled data, entailing a sophisticated multi-stage pipeline that
demands substantial human effort and incurs high training costs. Due to limited
prompting plans, these methods typically produce brief, information-poor, and
context-incoherent animations. To overcome these limitations and automate the
animation process, we pioneer the introduction of large multimodal models
(LMMs) as the core processor to build an autonomous animation-making agent,
named Anim-Director. This agent mainly harnesses the advanced understanding and
reasoning capabilities of LMMs and generative AI tools to create animated
videos from concise narratives or simple instructions. Specifically, it
operates in three main stages: Firstly, the Anim-Director generates a coherent
storyline from user inputs, followed by a detailed director's script that
encompasses settings of character profiles and interior/exterior descriptions,
and context-coherent scene descriptions that include appearing characters,
interiors or exteriors, and scene events. Secondly, we employ LMMs with the
image generation tool to produce visual images of settings and scenes. These
images are designed to maintain visual consistency across different scenes
using a visual-language prompting method that combines scene descriptions and
images of the appearing character and setting. Thirdly, scene images serve as
the foundation for producing animated videos, with LMMs generating prompts to
guide this process. The whole process is notably autonomous without manual
intervention, as the LMMs interact seamlessly with generative tools to generate
prompts, evaluate visual quality, and select the best one to optimize the final
output.
Anim-Director, an autonomous animation-making agent, uses large multimodal models to create consistent and coherent animations from concise narratives.
large multimodal models Anim-Director generative AI tools image generation visual-language prompting method animated videos scene descriptions visual consistency generative tools