VFX Creator: Animated Visual Effect Generation with Controllable
  Diffusion Transformer
Crafting magic and illusions is one of the most thrilling aspects of
filmmaking, with visual effects (VFX) serving as the powerhouse behind
unforgettable cinematic experiences. While recent advances in generative
artificial intelligence have driven progress in generic image and video
synthesis, the domain of controllable VFX generation remains relatively
underexplored. In this work, we propose a novel paradigm for animated VFX
generation as image animation, where dynamic effects are generated from
user-friendly textual descriptions and static reference images.
  Our work makes two primary contributions: (i) Open-VFX, the first
high-quality VFX video dataset spanning 15 diverse effect categories, annotated
with textual descriptions, instance segmentation masks for spatial
conditioning, and start-end timestamps for temporal control. (ii) VFX Creator,
a simple yet effective controllable VFX generation framework based on a Video
Diffusion Transformer. The model incorporates a spatial and temporal
controllable LoRA adapter, requiring minimal training videos. Specifically, a
plug-and-play mask control module enables instance-level spatial manipulation,
while tokenized start-end motion timestamps embedded in the diffusion process,
alongside the text encoder, allow precise temporal control over effect timing
and pace.
  Extensive experiments on the Open-VFX test set demonstrate the superiority of
the proposed system in generating realistic and dynamic effects, achieving
state-of-the-art performance and generalization ability in both spatial and
temporal controllability. Furthermore, we introduce a specialized metric to
evaluate the precision of temporal control. By bridging traditional VFX
techniques with generative approaches, VFX Creator unlocks new possibilities
for efficient and high-quality video effect generation, making advanced VFX
accessible to a broader audience.
67ab5cc2b8d7fe3b96361e31
Xinyu Liu
67ab5cc2b8d7fe3b96361e32
Ailing Zeng
extracted_pending
2025-02-11T14:21:00.414Z
665030b8c08859923b274a55
/avatars/06a3f3cd533caa459f3da1bb5e0f4c1f.svg
aznukad
user
alkxncda
67ab5cc2b8d7fe3b96361e33
Wei Xue
extracted_pending
2025-02-11T14:21:00.414Z
6628adb14277eae0da5eee28
/avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
HKUST Audio
user
HKUST-Audio
67ab5cc2b8d7fe3b96361e34
Harry Yang
67ab5cc2b8d7fe3b96361e35
Wenhan Luo
67ab5cc2b8d7fe3b96361e36
Qifeng Liu
67ab5cc2b8d7fe3b96361e37
Yike Guo
VFX Creator: Animated Visual Effect Generation with Controllable
  Diffusion Transformer
Crafting magic and illusions is one of the most thrilling aspects of
filmmaking, with visual effects (VFX) serving as the powerhouse behind
unforgettable cinematic experiences. While recent advances in generative
artificial intelligence have driven progress in generic image and video
synthesis, the domain of controllable VFX generation remains relatively
underexplored. In this work, we propose a novel paradigm for animated VFX
generation as image animation, where dynamic effects are generated from
user-friendly textual descriptions and static reference images.
  Our work makes two primary contributions: (i) Open-VFX, the first
high-quality VFX video dataset spanning 15 diverse effect categories, annotated
with textual descriptions, instance segmentation masks for spatial
conditioning, and start-end timestamps for temporal control. (ii) VFX Creator,
a simple yet effective controllable VFX generation framework based on a Video
Diffusion Transformer. The model incorporates a spatial and temporal
controllable LoRA adapter, requiring minimal training videos. Specifically, a
plug-and-play mask control module enables instance-level spatial manipulation,
while tokenized start-end motion timestamps embedded in the diffusion process,
alongside the text encoder, allow precise temporal control over effect timing
and pace.
  Extensive experiments on the Open-VFX test set demonstrate the superiority of
the proposed system in generating realistic and dynamic effects, achieving
state-of-the-art performance and generalization ability in both spatial and
temporal controllability. Furthermore, we introduce a specialized metric to
evaluate the precision of temporal control. By bridging traditional VFX
techniques with generative approaches, VFX Creator unlocks new possibilities
for efficient and high-quality video effect generation, making advanced VFX
accessible to a broader audience.
A novel VFX generation framework, VFX Creator, uses a Video Diffusion Transformer with spatial and temporal controllable LoRA adapters to generate dynamic effects from text and images, outperforming existing methods in realism and control.
Video Diffusion Transformer LoRA adapter instance-level spatial manipulation tokenized start-end motion timestamps diffusion process text encoder temporal control spatial condition temporal controllability metric for temporal control