WorldDreamer: Towards General World Models for Video Generation via
  Predicting Masked Tokens
World models play a crucial role in understanding and predicting the dynamics
of the world, which is essential for video generation. However, existing world
models are confined to specific scenarios such as gaming or driving, limiting
their ability to capture the complexity of general world dynamic environments.
Therefore, we introduce WorldDreamer, a pioneering world model to foster a
comprehensive comprehension of general world physics and motions, which
significantly enhances the capabilities of video generation. Drawing
inspiration from the success of large language models, WorldDreamer frames
world modeling as an unsupervised visual sequence modeling challenge. This is
achieved by mapping visual inputs to discrete tokens and predicting the masked
ones. During this process, we incorporate multi-modal prompts to facilitate
interaction within the world model. Our experiments show that WorldDreamer
excels in generating videos across different scenarios, including natural
scenes and driving environments. WorldDreamer showcases versatility in
executing tasks such as text-to-video conversion, image-tovideo synthesis, and
video editing. These results underscore WorldDreamer's effectiveness in
capturing dynamic elements within diverse general world environments.
65a9d8ec848a78d4da076152
Xiaofeng Wang
claimed_verified
2024-01-22T10:31:36.202Z
6426616ea5ec4a5cbc535634
https://cdn-avatars.huggingface.co/v1/production/uploads/6426616ea5ec4a5cbc535634/hH6JsxnXeakH3mBTeNNmO.jpeg
JeffWang
user
Jeff-Wang
65a9d8ec848a78d4da076153
Zheng Zhu
extracted_pending
2024-01-19T02:05:33.694Z
656e9b562cd7a3e348011d26
/avatars/bcca51bdc27c664f8f132420e6ed99fa.svg
Zheng Zhu
user
ZhengZhu
65a9d8ec848a78d4da076154
Guan Huang
65a9d8ec848a78d4da076155
Boyuan Wang
admin_assigned
2024-01-19T13:27:06.403Z
632c74564a4991e711531495
/avatars/f2c122c534127cf4c788d18156af2d98.svg
wang
user
boyuan
65a9d8ec848a78d4da076156
Xinze Chen
65a9d8ec848a78d4da076157
Jiwen Lu
WorldDreamer: Towards General World Models for Video Generation via
  Predicting Masked Tokens
World models play a crucial role in understanding and predicting the dynamics
of the world, which is essential for video generation. However, existing world
models are confined to specific scenarios such as gaming or driving, limiting
their ability to capture the complexity of general world dynamic environments.
Therefore, we introduce WorldDreamer, a pioneering world model to foster a
comprehensive comprehension of general world physics and motions, which
significantly enhances the capabilities of video generation. Drawing
inspiration from the success of large language models, WorldDreamer frames
world modeling as an unsupervised visual sequence modeling challenge. This is
achieved by mapping visual inputs to discrete tokens and predicting the masked
ones. During this process, we incorporate multi-modal prompts to facilitate
interaction within the world model. Our experiments show that WorldDreamer
excels in generating videos across different scenarios, including natural
scenes and driving environments. WorldDreamer showcases versatility in
executing tasks such as text-to-video conversion, image-tovideo synthesis, and
video editing. These results underscore WorldDreamer's effectiveness in
capturing dynamic elements within diverse general world environments.
WorldDreamer, a world model inspired by large language models, excels at unsupervised visual sequence modeling, generating videos in various scenarios with tasks such as text-to-video conversion and image-to-video synthesis.
world models visual sequence modeling discrete tokens multi-modal prompts text-to-video conversion image-to-video synthesis video editing