CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Recently video diffusion models have emerged as expressive generative tools
for high-quality video content creation readily available to general users.
However, these models often do not offer precise control over camera poses for
video generation, limiting the expression of cinematic language and user
control. To address this issue, we introduce CamCo, which allows fine-grained
Camera pose Control for image-to-video generation. We equip a pre-trained
image-to-video generator with accurately parameterized camera pose input using
Pl\"ucker coordinates. To enhance 3D consistency in the videos produced, we
integrate an epipolar attention module in each attention block that enforces
epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on
real-world videos with camera poses estimated through structure-from-motion
algorithms to better synthesize object motion. Our experiments show that CamCo
significantly improves 3D consistency and camera control capabilities compared
to previous models while effectively generating plausible object motion.
Project page: https://ir1d.github.io/CamCo/
665fe43a79373fff9c0f0781
Dejia Xu
admin_assigned
2024-06-05T10:45:22.237Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
665fe43a79373fff9c0f0782
Weili Nie
admin_assigned
2024-06-05T10:45:28.607Z
64c1a69e226e016da8450ae2
/avatars/54c161e8b8543244ed13cbe47017624e.svg
Weili Nie
user
xiaoli08
665fe43a79373fff9c0f0783
Chao Liu
admin_assigned
2024-06-05T10:45:54.513Z
63420d0706a304bdc4365aac
/avatars/06fe5b60a197ddf13538dde063b1eb0a.svg
Chao Liu
user
asroy
665fe43a79373fff9c0f0784
Sifei Liu
admin_assigned
2024-06-05T10:46:38.560Z
62fab69f8cd542e895bafd6e
/avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
Sifei Liu
user
zwrq
665fe43a79373fff9c0f0785
Jan Kautz
665fe43a79373fff9c0f0786
Zhangyang Wang
665fe43a79373fff9c0f0787
Arash Vahdat
admin_assigned
2024-06-05T10:47:00.058Z
62f6956cd3bdacb7eec02920
/avatars/b22db0823311f866c00db2efc4b9f814.svg
Arash Vahdat
user
avahdat
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Recently video diffusion models have emerged as expressive generative tools
for high-quality video content creation readily available to general users.
However, these models often do not offer precise control over camera poses for
video generation, limiting the expression of cinematic language and user
control. To address this issue, we introduce CamCo, which allows fine-grained
Camera pose Control for image-to-video generation. We equip a pre-trained
image-to-video generator with accurately parameterized camera pose input using
Pl\"ucker coordinates. To enhance 3D consistency in the videos produced, we
integrate an epipolar attention module in each attention block that enforces
epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on
real-world videos with camera poses estimated through structure-from-motion
algorithms to better synthesize object motion. Our experiments show that CamCo
significantly improves 3D consistency and camera control capabilities compared
to previous models while effectively generating plausible object motion.
Project page: https://ir1d.github.io/CamCo/
CamCo enhances camera pose control in video generation by using Plücker coordinates and an epipolar attention module, improving 3D consistency and object motion synthesis.
video diffusion models image-to-video generator Plücker coordinates epipolar attention module epipolar constraints feature maps structure-from-motion 3D consistency camera control object motion synthesis