Image Conductor: Precision Control for Interactive Video Synthesis
Filmmaking and animation production often require sophisticated techniques
for coordinating camera transitions and object movements, typically involving
labor-intensive real-world capturing. Despite advancements in generative AI for
video creation, achieving precise control over motion for interactive video
asset generation remains challenging. To this end, we propose Image Conductor,
a method for precise control of camera transitions and object movements to
generate video assets from a single image. An well-cultivated training strategy
is proposed to separate distinct camera and object motion by camera LoRA
weights and object LoRA weights. To further address cinematographic variations
from ill-posed trajectories, we introduce a camera-free guidance technique
during inference, enhancing object movements while eliminating camera
transitions. Additionally, we develop a trajectory-oriented video motion data
curation pipeline for training. Quantitative and qualitative experiments
demonstrate our method's precision and fine-grained control in generating
motion-controllable videos from images, advancing the practical application of
interactive video synthesis. Project webpage available at
https://liyaowei-stu.github.io/project/ImageConductor/
667abc244ede40ecf16d00a9
Yaowei Li
admin_assigned
2024-06-26T13:16:27.685Z
6362801380c1a705a6ea54ac
/avatars/041ad5abf9be42e336938f51ebb8746c.svg
Yaowei Li
user
Yw22
667abc244ede40ecf16d00aa
Xintao Wang
admin_assigned
2024-06-26T13:16:34.274Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
667abc244ede40ecf16d00ab
Zhaoyang Zhang
admin_assigned
2024-06-26T13:17:48.987Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
667abc244ede40ecf16d00ac
Zhouxia Wang
admin_assigned
2024-06-26T13:18:00.332Z
64db92a5858f8a41c11669b7
/avatars/e834d8f1d4781e3bb0b5d6d25b3b3505.svg
Zhouxia Wang
user
wzhouxiff
667abc244ede40ecf16d00ad
Ziyang Yuan
667abc244ede40ecf16d00ae
Liangbin Xie
admin_assigned
2024-06-26T13:18:14.101Z
6315dcb493fdd4a5633f7b30
/avatars/c58d228dbdf4f1edb5aa4e635dacb51d.svg
Liangbin Xie
user
Liangbin
667abc244ede40ecf16d00af
Yuexian Zou
667abc244ede40ecf16d00b0
Ying Shan
Image Conductor: Precision Control for Interactive Video Synthesis
Filmmaking and animation production often require sophisticated techniques
for coordinating camera transitions and object movements, typically involving
labor-intensive real-world capturing. Despite advancements in generative AI for
video creation, achieving precise control over motion for interactive video
asset generation remains challenging. To this end, we propose Image Conductor,
a method for precise control of camera transitions and object movements to
generate video assets from a single image. An well-cultivated training strategy
is proposed to separate distinct camera and object motion by camera LoRA
weights and object LoRA weights. To further address cinematographic variations
from ill-posed trajectories, we introduce a camera-free guidance technique
during inference, enhancing object movements while eliminating camera
transitions. Additionally, we develop a trajectory-oriented video motion data
curation pipeline for training. Quantitative and qualitative experiments
demonstrate our method's precision and fine-grained control in generating
motion-controllable videos from images, advancing the practical application of
interactive video synthesis. Project webpage available at
https://liyaowei-stu.github.io/project/ImageConductor/
Image Conductor generates motion-controllable videos from a single image using separate camera and object motion training strategies, enhancing precision and eliminating unnecessary camera transitions.
camera LoRA object LoRA camera-free guidance technique trajectory-oriented video motion data curation motion-controllable videos