Identity-Preserving Text-to-Video Generation by Frequency Decomposition
Identity-preserving text-to-video (IPT2V) generation aims to create
high-fidelity videos with consistent human identity. It is an important task in
video generation but remains an open problem for generative models. This paper
pushes the technical frontier of IPT2V in two directions that have not been
resolved in literature: (1) A tuning-free pipeline without tedious case-by-case
finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based
control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V
model to keep human identity consistent in the generated video. Inspired by
prior findings in frequency analysis of diffusion transformers, it employs
identity-control signals in the frequency domain, where facial features can be
decomposed into low-frequency global features and high-frequency intrinsic
features. First, from a low-frequency perspective, we introduce a global facial
extractor, which encodes reference images and facial key points into a latent
space, generating features enriched with low-frequency information. These
features are then integrated into shallow layers of the network to alleviate
training challenges associated with DiT. Second, from a high-frequency
perspective, we design a local facial extractor to capture high-frequency
details and inject them into transformer blocks, enhancing the model's ability
to preserve fine-grained features. We propose a hierarchical training strategy
to leverage frequency information for identity preservation, transforming a
vanilla pre-trained video generation model into an IPT2V model. Extensive
experiments demonstrate that our frequency-aware heuristic scheme provides an
optimal control solution for DiT-based models. Thanks to this scheme, our
ConsisID generates high-quality, identity-preserving videos, making strides
towards more effective IPT2V.
674698d29c25e8cefac0250f
Shenghai Yuan
claimed_verified
2024-11-28T09:19:50.317Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
user
BestWishYsh
674698d29c25e8cefac02510
Jinfa Huang
admin_assigned
2024-11-28T11:36:16.287Z
63f37af60be81bdc5d92eebb
/avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
Huang
user
Jinfa
674698d29c25e8cefac02511
Xianyi He
admin_assigned
2024-11-29T09:17:05.382Z
6688f94517f8d8f81c2def5f
/avatars/1064ccd62539a8ca2ec86fc87c818296.svg
hexianyi
user
pkuhexianyi
674698d29c25e8cefac02512
Yunyuan Ge
claimed_verified
2024-11-29T10:08:15.927Z
646de6402fd5a8eb8c518aa6
https://cdn-avatars.huggingface.co/v1/production/uploads/646de6402fd5a8eb8c518aa6/HYWb8-fT1kTm-ROBr1-0X.jpeg
yunyangge
user
yunyangge
674698d29c25e8cefac02513
Yujun Shi
674698d29c25e8cefac02514
Liuhan Chen
admin_assigned
2024-11-28T11:35:39.975Z
65c0588569429d85dc594a85
/avatars/822203d982519b999247a53235e30c39.svg
liuhan
user
LiuhanChen
674698d29c25e8cefac02515
Jiebo Luo
674698d29c25e8cefac02516
Li Yuan
claimed_verified
2025-04-04T07:11:13.193Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
Identity-Preserving Text-to-Video Generation by Frequency Decomposition
Identity-preserving text-to-video (IPT2V) generation aims to create
high-fidelity videos with consistent human identity. It is an important task in
video generation but remains an open problem for generative models. This paper
pushes the technical frontier of IPT2V in two directions that have not been
resolved in literature: (1) A tuning-free pipeline without tedious case-by-case
finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based
control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V
model to keep human identity consistent in the generated video. Inspired by
prior findings in frequency analysis of diffusion transformers, it employs
identity-control signals in the frequency domain, where facial features can be
decomposed into low-frequency global features and high-frequency intrinsic
features. First, from a low-frequency perspective, we introduce a global facial
extractor, which encodes reference images and facial key points into a latent
space, generating features enriched with low-frequency information. These
features are then integrated into shallow layers of the network to alleviate
training challenges associated with DiT. Second, from a high-frequency
perspective, we design a local facial extractor to capture high-frequency
details and inject them into transformer blocks, enhancing the model's ability
to preserve fine-grained features. We propose a hierarchical training strategy
to leverage frequency information for identity preservation, transforming a
vanilla pre-trained video generation model into an IPT2V model. Extensive
experiments demonstrate that our frequency-aware heuristic scheme provides an
optimal control solution for DiT-based models. Thanks to this scheme, our
ConsisID generates high-quality, identity-preserving videos, making strides
towards more effective IPT2V.
ConsisID is a tuning-free, DiT-based IPT2V model that uses frequency-aware identity control to generate high-quality, identity-preserving videos by decomposing facial features into low and high-frequency elements.
DiT-based IPT2V identity-preserving frequency-aware heuristic control global facial extractor low-frequency latent space local facial extractor high-frequency hierarchical training strategy video generation