Magic-Me: Identity-Specific Video Customized Diffusion
Creating content for a specific identity (ID) has shown significant interest
in the field of generative models. In the field of text-to-image generation
(T2I), subject-driven content generation has achieved great progress with the
ID in the images controllable. However, extending it to video generation is not
well explored. In this work, we propose a simple yet effective subject identity
controllable video generation framework, termed Video Custom Diffusion (VCD).
With a specified subject ID defined by a few images, VCD reinforces the
identity information extraction and injects frame-wise correlation at the
initialization stage for stable video outputs with identity preserved to a
large extent. To achieve this, we propose three novel components that are
essential for high-quality ID preservation: 1) an ID module trained with the
cropped identity by prompt-to-segmentation to disentangle the ID information
and the background noise for more accurate ID token learning; 2) a
text-to-video (T2V) VCD module with 3D Gaussian Noise Prior for better
inter-frame consistency and 3) video-to-video (V2V) Face VCD and Tiled VCD
modules to deblur the face and upscale the video for higher resolution.
  Despite its simplicity, we conducted extensive experiments to verify that VCD
is able to generate stable and high-quality videos with better ID over the
selected strong baselines. Besides, due to the transferability of the ID
module, VCD is also working well with finetuned text-to-image models available
publically, further improving its usability. The codes are available at
https://github.com/Zhen-Dong/Magic-Me.
65cd79e23f97d5bd5d51c04a
Ze Ma
65cd79e23f97d5bd5d51c04b
Daquan Zhou
admin_assigned
2024-02-15T08:50:24.450Z
63fe0b160c1bbe8e29d2dd32
/avatars/bc574036287170a77057893efaa48e2d.svg
Zhou
user
DaQuan21
65cd79e23f97d5bd5d51c04c
Chun-Hsiao Yeh
admin_assigned
2024-02-15T08:50:31.526Z
6499eca0685215f7247bd5ce
/avatars/b6fea0c33c3c930c7314b99b414072a9.svg
Chun-Hsiao Yeh
user
danielchyeh
65cd79e23f97d5bd5d51c04d
Xue-She Wang
65cd79e23f97d5bd5d51c04e
Xiuyu Li
admin_assigned
2024-02-15T08:50:49.215Z
644570ba2d91b15b4c7f6311
/avatars/d5e66012066d0c330b8f23718b1499d8.svg
Xiuyu Li
user
xiuyul
65cd79e23f97d5bd5d51c04f
Huanrui Yang
65cd79e23f97d5bd5d51c050
Zhen Dong
claimed_verified
2024-02-15T08:26:22.160Z
643ba2f725681c3afaa8f05e
https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
Zhen Dong
user
zhendongucb
65cd79e23f97d5bd5d51c051
Kurt Keutzer
admin_assigned
2024-02-15T08:51:04.156Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
65cd79e23f97d5bd5d51c052
Jiashi Feng
Magic-Me: Identity-Specific Video Customized Diffusion
Creating content for a specific identity (ID) has shown significant interest
in the field of generative models. In the field of text-to-image generation
(T2I), subject-driven content generation has achieved great progress with the
ID in the images controllable. However, extending it to video generation is not
well explored. In this work, we propose a simple yet effective subject identity
controllable video generation framework, termed Video Custom Diffusion (VCD).
With a specified subject ID defined by a few images, VCD reinforces the
identity information extraction and injects frame-wise correlation at the
initialization stage for stable video outputs with identity preserved to a
large extent. To achieve this, we propose three novel components that are
essential for high-quality ID preservation: 1) an ID module trained with the
cropped identity by prompt-to-segmentation to disentangle the ID information
and the background noise for more accurate ID token learning; 2) a
text-to-video (T2V) VCD module with 3D Gaussian Noise Prior for better
inter-frame consistency and 3) video-to-video (V2V) Face VCD and Tiled VCD
modules to deblur the face and upscale the video for higher resolution.
  Despite its simplicity, we conducted extensive experiments to verify that VCD
is able to generate stable and high-quality videos with better ID over the
selected strong baselines. Besides, due to the transferability of the ID
module, VCD is also working well with finetuned text-to-image models available
publically, further improving its usability. The codes are available at
https://github.com/Zhen-Dong/Magic-Me.
A framework named Video Custom Diffusion (VCD) is proposed for subject identity controllable video generation, achieving stable and high-quality outputs with better identity preservation compared to existing methods.
Video Custom Diffusion VCD ID module prompt-to-segmentation ID token learning text-to-video T2V 3D Gaussian Noise Prior video-to-video V2V.Face VCD Tiled VCD deblur upscale