CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for
  Zero-Shot Customized Video Diffusion Transformers
Customized generation has achieved significant progress in image synthesis,
yet personalized video generation remains challenging due to temporal
inconsistencies and quality degradation. In this paper, we introduce
CustomVideoX, an innovative framework leveraging the video diffusion
transformer for personalized video generation from a reference image.
CustomVideoX capitalizes on pre-trained video networks by exclusively training
the LoRA parameters to extract reference features, ensuring both efficiency and
adaptability. To facilitate seamless interaction between the reference image
and video content, we propose 3D Reference Attention, which enables direct and
simultaneous engagement of reference image features with all video frames
across spatial and temporal dimensions. To mitigate the excessive influence of
reference image features and textual guidance on generated video content during
inference, we implement the Time-Aware Reference Attention Bias (TAB) strategy,
dynamically modulating reference bias over different time steps. Additionally,
we introduce the Entity Region-Aware Enhancement (ERAE) module, aligning highly
activated regions of key entity tokens with reference feature injection by
adjusting attention bias. To thoroughly evaluate personalized video generation,
we establish a new benchmark, VideoBench, comprising over 50 objects and 100
prompts for extensive assessment. Experimental results show that CustomVideoX
significantly outperforms existing methods in terms of video consistency and
quality.
67aae4128d478dcb4b39a097
D. She
67aae4128d478dcb4b39a098
Mushui Liu
67aae4128d478dcb4b39a099
Jingxuan Pang
67aae4128d478dcb4b39a09a
Jin Wang
67aae4128d478dcb4b39a09b
Zhen Yang
67aae4128d478dcb4b39a09c
Wanggui He
67aae4128d478dcb4b39a09d
Guanghao Zhang
67aae4128d478dcb4b39a09e
Yi Wang
67aae4128d478dcb4b39a09f
Qihan Huang
67aae4128d478dcb4b39a0a0
Haobin Tang
67aae4128d478dcb4b39a0a1
Yunlong Yu
67aae4128d478dcb4b39a0a2
Siming Fu
CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for
  Zero-Shot Customized Video Diffusion Transformers
Customized generation has achieved significant progress in image synthesis,
yet personalized video generation remains challenging due to temporal
inconsistencies and quality degradation. In this paper, we introduce
CustomVideoX, an innovative framework leveraging the video diffusion
transformer for personalized video generation from a reference image.
CustomVideoX capitalizes on pre-trained video networks by exclusively training
the LoRA parameters to extract reference features, ensuring both efficiency and
adaptability. To facilitate seamless interaction between the reference image
and video content, we propose 3D Reference Attention, which enables direct and
simultaneous engagement of reference image features with all video frames
across spatial and temporal dimensions. To mitigate the excessive influence of
reference image features and textual guidance on generated video content during
inference, we implement the Time-Aware Reference Attention Bias (TAB) strategy,
dynamically modulating reference bias over different time steps. Additionally,
we introduce the Entity Region-Aware Enhancement (ERAE) module, aligning highly
activated regions of key entity tokens with reference feature injection by
adjusting attention bias. To thoroughly evaluate personalized video generation,
we establish a new benchmark, VideoBench, comprising over 50 objects and 100
prompts for extensive assessment. Experimental results show that CustomVideoX
significantly outperforms existing methods in terms of video consistency and
quality.
CustomVideoX, a framework using video diffusion transformer and pre-trained video networks, achieves improved personalized video generation by integrating reference images efficiently and dynamically adjusting attention bias.
video diffusion transformer pre-trained video networks LoRA parameters 3D Reference Attention Time-Aware Reference Attention Bias TAB strategy Entity Region-Aware Enhancement ERAE module VideoBench video consistency quality