SEED-Story: Multimodal Long Story Generation with Large Language Model
With the remarkable advancements in image generation and open-form text
generation, the creation of interleaved image-text content has become an
increasingly intriguing field. Multimodal story generation, characterized by
producing narrative texts and vivid images in an interleaved manner, has
emerged as a valuable and practical task with broad applications. However, this
task poses significant challenges, as it necessitates the comprehension of the
complex interplay between texts and images, and the ability to generate long
sequences of coherent, contextually relevant texts and visuals. In this work,
we propose SEED-Story, a novel method that leverages a Multimodal Large
Language Model (MLLM) to generate extended multimodal stories. Our model, built
upon the powerful comprehension capability of MLLM, predicts text tokens as
well as visual tokens, which are subsequently processed with an adapted visual
de-tokenizer to produce images with consistent characters and styles. We
further propose multimodal attention sink mechanism to enable the generation of
stories with up to 25 sequences (only 10 for training) in a highly efficient
autoregressive manner. Additionally, we present a large-scale and
high-resolution dataset named StoryStream for training our model and
quantitatively evaluating the task of multimodal story generation in various
aspects.
669091001e3f5b16ce22cce6
Shuai Yang
claimed_verified
2024-07-12T16:03:30.318Z
634ce90e741a5e37886a19e3
/avatars/0d1579039136b37db5b67282b0a34c33.svg
Syang
user
Andyson
669091001e3f5b16ce22cce7
Yuying Ge
admin_assigned
2024-07-12T10:22:50.689Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
669091001e3f5b16ce22cce8
Yang Li
669091001e3f5b16ce22cce9
Yukang Chen
admin_assigned
2024-07-12T10:25:40.867Z
65b31d77f32c79fea79ba4d1
/avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
Yukang Chen
user
Yukang2
669091001e3f5b16ce22ccea
Yixiao Ge
admin_assigned
2024-07-12T10:25:34.813Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
669091001e3f5b16ce22cceb
Ying Shan
admin_assigned
2024-07-12T16:09:04.359Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
669091001e3f5b16ce22ccec
Yingcong Chen
admin_assigned
2024-07-12T10:25:23.622Z
655cba1d87b67834000590e8
/avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
Yingcong Chen
user
yingcongchen
SEED-Story: Multimodal Long Story Generation with Large Language Model
With the remarkable advancements in image generation and open-form text
generation, the creation of interleaved image-text content has become an
increasingly intriguing field. Multimodal story generation, characterized by
producing narrative texts and vivid images in an interleaved manner, has
emerged as a valuable and practical task with broad applications. However, this
task poses significant challenges, as it necessitates the comprehension of the
complex interplay between texts and images, and the ability to generate long
sequences of coherent, contextually relevant texts and visuals. In this work,
we propose SEED-Story, a novel method that leverages a Multimodal Large
Language Model (MLLM) to generate extended multimodal stories. Our model, built
upon the powerful comprehension capability of MLLM, predicts text tokens as
well as visual tokens, which are subsequently processed with an adapted visual
de-tokenizer to produce images with consistent characters and styles. We
further propose multimodal attention sink mechanism to enable the generation of
stories with up to 25 sequences (only 10 for training) in a highly efficient
autoregressive manner. Additionally, we present a large-scale and
high-resolution dataset named StoryStream for training our model and
quantitatively evaluating the task of multimodal story generation in various
aspects.
SEED-Story uses a Multimodal Large Language Model with multimodal attention to generate extended multimodal stories from interleaved image-text sequences.
Multimodal Large Language Model MLLM text tokens visual tokens visual de-tokenizer multimodal attention sink mechanism autoregressive generation StoryStream dataset