SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
Methods for image-to-video generation have achieved impressive,
photo-realistic quality. However, adjusting specific elements in generated
videos, such as object motion or camera movement, is often a tedious process of
trial and error, e.g., involving re-generating videos with different random
seeds. Recent techniques address this issue by fine-tuning a pre-trained model
to follow conditioning signals, such as bounding boxes or point trajectories.
Yet, this fine-tuning procedure can be computationally expensive, and it
requires datasets with annotated object motion, which can be difficult to
procure. In this work, we introduce SG-I2V, a framework for controllable
image-to-video generation that is self-guidedx2013offering
zero-shot control by relying solely on the knowledge present in a pre-trained
image-to-video diffusion model without the need for fine-tuning or external
knowledge. Our zero-shot method outperforms unsupervised baselines while being
competitive with supervised models in terms of visual quality and motion
fidelity.
672d851d14742348552868ae
Koichi Namekata
claimed_verified
2024-11-15T09:31:25.687Z
65aec2b6101482afcc3126f2
https://cdn-avatars.huggingface.co/v1/production/uploads/65aec2b6101482afcc3126f2/IEKT6u1Upx2clt7qhhoUB.jpeg
Koichi Namekata
user
kmcode
672d851d14742348552868af
Sherwin Bahmani
672d851d14742348552868b0
Ziyi Wu
claimed_verified
2024-12-06T09:09:59.064Z
666bb5a310073a7de7ba0d8d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Z-yB3jbYIjhZQ9WwJhleq.jpeg
Ziyi Wu
user
Wuziyi616
672d851d14742348552868b1
Yash Kant
claimed_verified
2025-02-13T08:26:49.298Z
638546d0a179f856005ae310
/avatars/f7f2fface336c8e168a1daaf9fd4d40c.svg
yashkant
user
yashkant
672d851d14742348552868b2
Igor Gilitschenski
672d851d14742348552868b3
David B. Lindell
SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
Methods for image-to-video generation have achieved impressive,
photo-realistic quality. However, adjusting specific elements in generated
videos, such as object motion or camera movement, is often a tedious process of
trial and error, e.g., involving re-generating videos with different random
seeds. Recent techniques address this issue by fine-tuning a pre-trained model
to follow conditioning signals, such as bounding boxes or point trajectories.
Yet, this fine-tuning procedure can be computationally expensive, and it
requires datasets with annotated object motion, which can be difficult to
procure. In this work, we introduce SG-I2V, a framework for controllable
image-to-video generation that is self-guidedx2013offering
zero-shot control by relying solely on the knowledge present in a pre-trained
image-to-video diffusion model without the need for fine-tuning or external
knowledge. Our zero-shot method outperforms unsupervised baselines while being
competitive with supervised models in terms of visual quality and motion
fidelity.
SG-I2V is a self-guided framework for image-to-video generation that achieves zero-shot control and competitive visual quality and motion fidelity without fine-tuning or external knowledge.
image-to-video generation diffusion model zero-shot control visual quality motion fidelity