VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video
  Generation
Recent image-to-video generation methods have demonstrated success in
enabling control over one or two visual elements, such as camera trajectory or
object motion. However, these methods are unable to offer control over multiple
visual elements due to limitations in data and network efficacy. In this paper,
we introduce VidCRAFT3, a novel framework for precise image-to-video generation
that enables control over camera motion, object motion, and lighting direction
simultaneously. To better decouple control over each visual element, we propose
the Spatial Triple-Attention Transformer, which integrates lighting direction,
text, and image in a symmetric way. Since most real-world video datasets lack
lighting annotations, we construct a high-quality synthetic video dataset, the
VideoLightingDirection (VLD) dataset. This dataset includes lighting direction
annotations and objects of diverse appearance, enabling VidCRAFT3 to
effectively handle strong light transmission and reflection effects.
Additionally, we propose a three-stage training strategy that eliminates the
need for training data annotated with multiple visual elements (camera motion,
object motion, and lighting direction) simultaneously. Extensive experiments on
benchmark datasets demonstrate the efficacy of VidCRAFT3 in producing
high-quality video content, surpassing existing state-of-the-art methods in
terms of control granularity and visual coherence. All code and data will be
publicly available. Project page: https://sixiaozheng.github.io/VidCRAFT3/.
67ac21acaa680a0f8782d273
Sixiao Zheng
claimed_verified
2025-03-14T08:59:27.188Z
6528fe394513680346a500ed
/avatars/3021a3be7d900e3928cdc375e2a2bf09.svg
Sixiao Zheng
user
sxzheng
67ac21acaa680a0f8782d274
Zimian Peng
67ac21acaa680a0f8782d275
Yanpeng Zhou
67ac21acaa680a0f8782d276
Yi Zhu
67ac21acaa680a0f8782d277
Hang Xu
67ac21acaa680a0f8782d278
Xiangru Huang
67ac21acaa680a0f8782d279
Yanwei Fu
VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video
  Generation
Recent image-to-video generation methods have demonstrated success in
enabling control over one or two visual elements, such as camera trajectory or
object motion. However, these methods are unable to offer control over multiple
visual elements due to limitations in data and network efficacy. In this paper,
we introduce VidCRAFT3, a novel framework for precise image-to-video generation
that enables control over camera motion, object motion, and lighting direction
simultaneously. To better decouple control over each visual element, we propose
the Spatial Triple-Attention Transformer, which integrates lighting direction,
text, and image in a symmetric way. Since most real-world video datasets lack
lighting annotations, we construct a high-quality synthetic video dataset, the
VideoLightingDirection (VLD) dataset. This dataset includes lighting direction
annotations and objects of diverse appearance, enabling VidCRAFT3 to
effectively handle strong light transmission and reflection effects.
Additionally, we propose a three-stage training strategy that eliminates the
need for training data annotated with multiple visual elements (camera motion,
object motion, and lighting direction) simultaneously. Extensive experiments on
benchmark datasets demonstrate the efficacy of VidCRAFT3 in producing
high-quality video content, surpassing existing state-of-the-art methods in
terms of control granularity and visual coherence. All code and data will be
publicly available. Project page: https://sixiaozheng.github.io/VidCRAFT3/.
VidCRAFT3, a novel framework, uses a Spatial Triple-Attention Transformer and a synthetic dataset to enable simultaneous control over camera motion, object motion, and lighting direction, producing high-quality video content.
Spatial Triple-Attention Transformer VideoLightingDirection (VLD) dataset control granularity visual coherence