Dynamic Concepts Personalization from Single Videos
Personalizing generative text-to-image models has seen remarkable progress,
but extending this personalization to text-to-video models presents unique
challenges. Unlike static concepts, personalizing text-to-video models has the
potential to capture dynamic concepts, i.e., entities defined not only by their
appearance but also by their motion. In this paper, we introduce
Set-and-Sequence, a novel framework for personalizing Diffusion Transformers
(DiTs)-based generative video models with dynamic concepts. Our approach
imposes a spatio-temporal weight space within an architecture that does not
explicitly separate spatial and temporal features. This is achieved in two key
stages. First, we fine-tune Low-Rank Adaptation (LoRA) layers using an
unordered set of frames from the video to learn an identity LoRA basis that
represents the appearance, free from temporal interference. In the second
stage, with the identity LoRAs frozen, we augment their coefficients with
Motion Residuals and fine-tune them on the full video sequence, capturing
motion dynamics. Our Set-and-Sequence framework results in a spatio-temporal
weight space that effectively embeds dynamic concepts into the video model's
output domain, enabling unprecedented editability and compositionality while
setting a new benchmark for personalizing dynamic concepts.
67b7f5ee8b3dff28b749be78
Rameen Abdal
admin_assigned
2025-02-21T15:00:23.357Z
630428fa7b50dd9d0a38cde0
/avatars/d1baf7fd17daf4be16ba5bd6cd4f2277.svg
Rameen Abdal
user
RameenAbdal
67b7f5ee8b3dff28b749be79
Or Patashnik
admin_assigned
2025-02-21T15:00:32.474Z
62853516e483e0d37b354ce1
https://cdn-avatars.huggingface.co/v1/production/uploads/62853516e483e0d37b354ce1/t5Tyd3E07w26B9Z3XpZWI.jpeg
Or Patashnik
user
orpatashnik
67b7f5ee8b3dff28b749be7a
Ivan Skorokhodov
admin_assigned
2025-02-21T15:00:39.038Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
67b7f5ee8b3dff28b749be7b
Willi Menapace
admin_assigned
2025-02-21T15:00:45.032Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
67b7f5ee8b3dff28b749be7c
Aliaksandr Siarohin
admin_assigned
2025-02-21T15:00:51.221Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
67b7f5ee8b3dff28b749be7d
Sergey Tulyakov
67b7f5ee8b3dff28b749be7e
Daniel Cohen-Or
admin_assigned
2025-02-21T15:01:00.737Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
67b7f5ee8b3dff28b749be7f
Kfir Aberman
admin_assigned
2025-02-21T15:01:06.434Z
64db29097266618e853dd6ec
https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
Kfir Aberman
user
kaberman
Dynamic Concepts Personalization from Single Videos
Personalizing generative text-to-image models has seen remarkable progress,
but extending this personalization to text-to-video models presents unique
challenges. Unlike static concepts, personalizing text-to-video models has the
potential to capture dynamic concepts, i.e., entities defined not only by their
appearance but also by their motion. In this paper, we introduce
Set-and-Sequence, a novel framework for personalizing Diffusion Transformers
(DiTs)-based generative video models with dynamic concepts. Our approach
imposes a spatio-temporal weight space within an architecture that does not
explicitly separate spatial and temporal features. This is achieved in two key
stages. First, we fine-tune Low-Rank Adaptation (LoRA) layers using an
unordered set of frames from the video to learn an identity LoRA basis that
represents the appearance, free from temporal interference. In the second
stage, with the identity LoRAs frozen, we augment their coefficients with
Motion Residuals and fine-tune them on the full video sequence, capturing
motion dynamics. Our Set-and-Sequence framework results in a spatio-temporal
weight space that effectively embeds dynamic concepts into the video model's
output domain, enabling unprecedented editability and compositionality while
setting a new benchmark for personalizing dynamic concepts.
The Set-and-Sequence framework personalizes Diffusion Transformers for generative video models by learning spatial identities and motion dynamics, significantly enhancing video editability and performance.
Diffusion Transformers Low-Rank Adaptation Motion Residuals spatio-temporal weight space identity LoRA spatial identities motion dynamics