Story-Adapter: A Training-free Iterative Framework for Long Story
  Visualization
Story visualization, the task of generating coherent images based on a
narrative, has seen significant advancements with the emergence of
text-to-image models, particularly diffusion models. However, maintaining
semantic consistency, generating high-quality fine-grained interactions, and
ensuring computational feasibility remain challenging, especially in long story
visualization (i.e., up to 100 frames). In this work, we propose a
training-free and computationally efficient framework, termed Story-Adapter, to
enhance the generative capability of long stories. Specifically, we propose an
iterative paradigm to refine each generated image, leveraging both the text
prompt and all generated images from the previous iteration. Central to our
framework is a training-free global reference cross-attention module, which
aggregates all generated images from the previous iteration to preserve
semantic consistency across the entire story, while minimizing computational
costs with global embeddings. This iterative process progressively optimizes
image generation by repeatedly incorporating text constraints, resulting in
more precise and fine-grained interactions. Extensive experiments validate the
superiority of Story-Adapter in improving both semantic consistency and
generative capability for fine-grained interactions, particularly in long story
scenarios. The project page and associated code can be accessed via
https://jwmao1.github.io/storyadapter .
670745bbd6f7382cc428eafc
Jiawei Mao
admin_assigned
2024-10-10T09:01:11.990Z
65f261116c60cd168b05433c
/avatars/0515bef9df2514707cbd3fc281891e2d.svg
Jiawei Mao
user
JohnWeck
670745bbd6f7382cc428eafd
Xiaoke Huang
claimed_verified
2024-10-10T08:01:17.167Z
63318b2349a9563915469f3b
https://cdn-avatars.huggingface.co/v1/production/uploads/63318b2349a9563915469f3b/zlbeB2997i8YkoyOTb9FL.jpeg
Xiaoke Huang
user
xk-huang
670745bbd6f7382cc428eafe
Yunfei Xie
claimed_verified
2024-10-11T07:44:16.578Z
65f6afa36ea40b9a29dd45ca
/avatars/a80a25fa70a89a3bf3d50ea2866f0df1.svg
Yunfei Xie
user
yunfeixie
670745bbd6f7382cc428eaff
Yuanqi Chang
670745bbd6f7382cc428eb00
Mude Hui
admin_assigned
2024-10-10T09:01:57.443Z
659f6f0de06dc8fc0e411224
/avatars/e16170f36b949c8697f277f6b52c51c3.svg
MudeHui
user
MudeHui
670745bbd6f7382cc428eb01
Bingjie Xu
670745bbd6f7382cc428eb02
Yuyin Zhou
admin_assigned
2024-10-10T09:02:11.011Z
66c7fb4ce2c92fe5b132f314
/avatars/22d915fa339a70803c5c748255250256.svg
Yuyin Zhou
user
RitaCoding
Story-Adapter: A Training-free Iterative Framework for Long Story
  Visualization
Story visualization, the task of generating coherent images based on a
narrative, has seen significant advancements with the emergence of
text-to-image models, particularly diffusion models. However, maintaining
semantic consistency, generating high-quality fine-grained interactions, and
ensuring computational feasibility remain challenging, especially in long story
visualization (i.e., up to 100 frames). In this work, we propose a
training-free and computationally efficient framework, termed Story-Adapter, to
enhance the generative capability of long stories. Specifically, we propose an
iterative paradigm to refine each generated image, leveraging both the text
prompt and all generated images from the previous iteration. Central to our
framework is a training-free global reference cross-attention module, which
aggregates all generated images from the previous iteration to preserve
semantic consistency across the entire story, while minimizing computational
costs with global embeddings. This iterative process progressively optimizes
image generation by repeatedly incorporating text constraints, resulting in
more precise and fine-grained interactions. Extensive experiments validate the
superiority of Story-Adapter in improving both semantic consistency and
generative capability for fine-grained interactions, particularly in long story
scenarios. The project page and associated code can be accessed via
https://jwmao1.github.io/storyadapter .
Story-Adapter is a training-free framework that enhances text-to-image models for long story visualization by iteratively refining images and maintaining semantic consistency through global reference cross-attention.
text-to-image models diffusion models semantic consistency fine-grained interactions computational feasibility iterative paradigm global reference cross-attention module global embeddings image generation text constraints