Ingredients: Blending Custom Photos with Video Diffusion Transformers
This paper presents a powerful framework to customize video creations by
incorporating multiple specific identity (ID) photos, with video diffusion
Transformers, referred to as Ingredients. Generally, our method
consists of three primary modules: (i) a facial extractor that
captures versatile and precise facial features for each human ID from both
global and local perspectives; (ii) a multi-scale projector that maps
face embeddings into the contextual space of image query in video diffusion
transformers; (iii) an ID router that dynamically combines and
allocates multiple ID embedding to the corresponding space-time regions.
Leveraging a meticulously curated text-video dataset and a multi-stage training
protocol, Ingredients demonstrates superior performance in turning
custom photos into dynamic and personalized video content. Qualitative
evaluations highlight the advantages of proposed method, positioning it as a
significant advancement toward more effective generative video control tools in
Transformer-based architecture, compared to existing methods. The data, code,
and model weights are publicly available at:
https://github.com/feizc/Ingredients.
677c162919230f94bdcc45e2
Zhengcong Fei
admin_assigned
2025-01-07T10:01:30.625Z
617ba1820e4237bd1731b867
/avatars/f9de06363e64bddd7dc977e96e85df8a.svg
zhengcong fei
user
onion
677c162919230f94bdcc45e3
Debang Li
admin_assigned
2025-01-07T10:01:40.288Z
65dc3a850af7e21ba40e939f
/avatars/e129c64617675edd05d4317d39604318.svg
Li
user
Debang
677c162919230f94bdcc45e4
Di Qiu
claimed_verified
2025-01-07T13:15:36.627Z
65bef422fdb8d33cefeaccc3
/avatars/d40b0d7dda21fa1a68c291d11bc357ec.svg
Qiu Di
user
diqiu7
677c162919230f94bdcc45e5
Changqian Yu
admin_assigned
2025-01-07T10:01:50.216Z
6419e0b3ed725fef6444f53a
https://cdn-avatars.huggingface.co/v1/production/uploads/6419e0b3ed725fef6444f53a/3ZzIafBJa50-gAu1PASuG.png
Yu Changqian
user
Changqian
677c162919230f94bdcc45e6
Mingyuan Fan
admin_assigned
2025-01-07T10:02:01.415Z
634672bfb7b4e71c7f45360f
/avatars/4b646fc3e271be90b9ec619d42ce3e99.svg
Fan Mingyuan
user
MichaelFan
Ingredients: Blending Custom Photos with Video Diffusion Transformers
This paper presents a powerful framework to customize video creations by
incorporating multiple specific identity (ID) photos, with video diffusion
Transformers, referred to as Ingredients. Generally, our method
consists of three primary modules: (i) a facial extractor that
captures versatile and precise facial features for each human ID from both
global and local perspectives; (ii) a multi-scale projector that maps
face embeddings into the contextual space of image query in video diffusion
transformers; (iii) an ID router that dynamically combines and
allocates multiple ID embedding to the corresponding space-time regions.
Leveraging a meticulously curated text-video dataset and a multi-stage training
protocol, Ingredients demonstrates superior performance in turning
custom photos into dynamic and personalized video content. Qualitative
evaluations highlight the advantages of proposed method, positioning it as a
significant advancement toward more effective generative video control tools in
Transformer-based architecture, compared to existing methods. The data, code,
and model weights are publicly available at:
https://github.com/feizc/Ingredients.
A framework called Ingredients uses video diffusion Transformers to create customized videos by integrating multiple identity photos through facial extraction, multi-scale projection, and ID routing.
video diffusion Transformers facial extractor multi-scale projector ID router contextual space image query space-time regions multi-stage training protocol generative video control tools