OmniCreator: Self-Supervised Unified Generation with Universal Editing
We introduce OmniCreator, a novel framework that can conduct text-prompted
unified (image+video) generation as well as editing all in one place.
OmniCreator acquires generative and universal editing capabilities in a
self-supervised manner, taking original text-video pairs as conditions while
utilizing the same video as a denoising target to learn the semantic
correspondence between video and text. During inference, when presented with a
text prompt and a video, OmniCreator is capable of generating a target that is
faithful to both, achieving a universal editing effect that is unconstrained as
opposed to existing editing work that primarily focuses on certain editing
types or relies on additional controls (e.g., structural conditions, attention
features, or DDIM inversion). On the other hand, when presented with a text
prompt only, OmniCreator becomes generative, producing high-quality video as a
result of the semantic correspondence learned. Importantly, we found that the
same capabilities extend to images as is, making OmniCreator a truly unified
framework. Further, due to the lack of existing generative video editing
benchmarks, we introduce the OmniBench-99 dataset, designed to evaluate the
performance of generative video editing models comprehensively. Extensive
experiments demonstrate that OmniCreator exhibits substantial superiority over
all other models.
674fe64737c700d35214dda6
Haodong Chen
claimed_verified
2024-12-04T09:44:20.090Z
6570450a78d7aca0c361a177
https://cdn-avatars.huggingface.co/v1/production/uploads/6570450a78d7aca0c361a177/z0GrnXEsjK2_G-hFfQhKv.jpeg
Harold Chen
user
Harold328
674fe64737c700d35214dda7
Lan Wang
674fe64737c700d35214dda8
Harry Yang
674fe64737c700d35214dda9
Ser-Nam Lim
admin_assigned
2024-12-04T12:00:17.400Z
65884498ce38d143c435d279
/avatars/96a917c40680a79a37dcb8fa75014c21.svg
Ser Nam Lim
user
sernam
OmniCreator: Self-Supervised Unified Generation with Universal Editing
We introduce OmniCreator, a novel framework that can conduct text-prompted
unified (image+video) generation as well as editing all in one place.
OmniCreator acquires generative and universal editing capabilities in a
self-supervised manner, taking original text-video pairs as conditions while
utilizing the same video as a denoising target to learn the semantic
correspondence between video and text. During inference, when presented with a
text prompt and a video, OmniCreator is capable of generating a target that is
faithful to both, achieving a universal editing effect that is unconstrained as
opposed to existing editing work that primarily focuses on certain editing
types or relies on additional controls (e.g., structural conditions, attention
features, or DDIM inversion). On the other hand, when presented with a text
prompt only, OmniCreator becomes generative, producing high-quality video as a
result of the semantic correspondence learned. Importantly, we found that the
same capabilities extend to images as is, making OmniCreator a truly unified
framework. Further, due to the lack of existing generative video editing
benchmarks, we introduce the OmniBench-99 dataset, designed to evaluate the
performance of generative video editing models comprehensively. Extensive
experiments demonstrate that OmniCreator exhibits substantial superiority over
all other models.
OmniCreator is a unified framework that conducts text-prompted video and image generation and editing, learning semantic correspondence self-supervised and outperforming existing models.
OmniCreator text-prompted unified generation editing self-supervised learning denoising semantic correspondence OmniBench-99 dataset video editing image generation