MultiCOIN: Multi-Modal COntrollable Video INbetweening
Video inbetweening creates smooth and natural transitions between two image
frames, making it an indispensable tool for video editing and long-form video
synthesis. Existing works in this domain are unable to generate large, complex,
or intricate motions. In particular, they cannot accommodate the versatility of
user intents and generally lack fine control over the details of intermediate
frames, leading to misalignment with the creative mind. To fill these gaps, we
introduce MultiCOIN, a video inbetweening framework that allows multi-modal
controls, including depth transition and layering, motion trajectories, text
prompts, and target regions for movement localization, while achieving a
balance between flexibility, ease of use, and precision for fine-grained video
interpolation. To achieve this, we adopt the Diffusion Transformer (DiT)
architecture as our video generative model, due to its proven capability to
generate high-quality long videos. To ensure compatibility between DiT and our
multi-modal controls, we map all motion controls into a common sparse and
user-friendly point-based representation as the video/noise input. Further, to
respect the variety of controls which operate at varying levels of granularity
and influence, we separate content controls and motion controls into two
branches to encode the required features before guiding the denoising process,
resulting in two generators, one for motion and the other for content. Finally,
we propose a stage-wise training strategy to ensure that our model learns the
multi-modal controls smoothly. Extensive qualitative and quantitative
experiments demonstrate that multi-modal controls enable a more dynamic,
customizable, and contextually accurate visual narrative.
68ee7ff97b0ac1ee3e5998b2
Maham Tanveer
claimed_verified
2025-10-15T03:13:43.408Z
6490dd690c18343a093dc718
https://cdn-avatars.huggingface.co/v1/production/uploads/6490dd690c18343a093dc718/RgzJyka25k4F_ovVQVkJD.png
Maham Tanveer
user
tmaham
68ee7ff97b0ac1ee3e5998b3
Yang Zhou
68ee7ff97b0ac1ee3e5998b4
Simon Niklaus
68ee7ff97b0ac1ee3e5998b5
Ali Mahdavi Amiri
68ee7ff97b0ac1ee3e5998b6
Hao Zhang
68ee7ff97b0ac1ee3e5998b7
Krishna Kumar Singh
68ee7ff97b0ac1ee3e5998b8
Nanxuan Zhao
MultiCOIN: Multi-Modal COntrollable Video INbetweening
Video inbetweening creates smooth and natural transitions between two image
frames, making it an indispensable tool for video editing and long-form video
synthesis. Existing works in this domain are unable to generate large, complex,
or intricate motions. In particular, they cannot accommodate the versatility of
user intents and generally lack fine control over the details of intermediate
frames, leading to misalignment with the creative mind. To fill these gaps, we
introduce MultiCOIN, a video inbetweening framework that allows multi-modal
controls, including depth transition and layering, motion trajectories, text
prompts, and target regions for movement localization, while achieving a
balance between flexibility, ease of use, and precision for fine-grained video
interpolation. To achieve this, we adopt the Diffusion Transformer (DiT)
architecture as our video generative model, due to its proven capability to
generate high-quality long videos. To ensure compatibility between DiT and our
multi-modal controls, we map all motion controls into a common sparse and
user-friendly point-based representation as the video/noise input. Further, to
respect the variety of controls which operate at varying levels of granularity
and influence, we separate content controls and motion controls into two
branches to encode the required features before guiding the denoising process,
resulting in two generators, one for motion and the other for content. Finally,
we propose a stage-wise training strategy to ensure that our model learns the
multi-modal controls smoothly. Extensive qualitative and quantitative
experiments demonstrate that multi-modal controls enable a more dynamic,
customizable, and contextually accurate visual narrative.
MultiCOIN, a video inbetweening framework using the Diffusion Transformer, enables multi-modal controls for precise and flexible video interpolation.
Diffusion Transformer DiT video inbetweening multi-modal controls depth transition layering motion trajectories text prompts target regions motion controls content controls denoising process stage-wise training