Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
Audio-video generation has often relied on complex multi-stage architectures
or sequential synthesis of sound and visuals. We introduce Ovi, a unified
paradigm for audio-video generation that models the two modalities as a single
generative process. By using blockwise cross-modal fusion of twin-DiT modules,
Ovi achieves natural synchronization and removes the need for separate
pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion
modeling, we initialize an audio tower with an architecture identical to that
of a strong pretrained video model. Trained from scratch on hundreds of
thousands of hours of raw audio, the audio tower learns to generate realistic
sound effects, as well as speech that conveys rich speaker identity and
emotion. Fusion is obtained by jointly training the identical video and audio
towers via blockwise exchange of timing (via scaled-RoPE embeddings) and
semantics (through bidirectional cross-attention) on a vast video corpus. Our
model enables cinematic storytelling with natural speech and accurate,
context-matched sound effects, producing movie-grade video clips. All the
demos, code and model weights are published at https://aaxwaz.github.io/Ovi
68df3d06df49fb0df1e03d05
Chetwin Low
68df3d06df49fb0df1e03d06
Weimin Wang
68df3d06df49fb0df1e03d07
Calder Katyal
claimed_verified
2025-10-03T12:29:39.790Z
6743e005ac35394749ca2aaf
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/A4856Ek1Cd3wumFwS1_Z3.png
Calder Katyal
user
CalderKat
Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
Audio-video generation has often relied on complex multi-stage architectures
or sequential synthesis of sound and visuals. We introduce Ovi, a unified
paradigm for audio-video generation that models the two modalities as a single
generative process. By using blockwise cross-modal fusion of twin-DiT modules,
Ovi achieves natural synchronization and removes the need for separate
pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion
modeling, we initialize an audio tower with an architecture identical to that
of a strong pretrained video model. Trained from scratch on hundreds of
thousands of hours of raw audio, the audio tower learns to generate realistic
sound effects, as well as speech that conveys rich speaker identity and
emotion. Fusion is obtained by jointly training the identical video and audio
towers via blockwise exchange of timing (via scaled-RoPE embeddings) and
semantics (through bidirectional cross-attention) on a vast video corpus. Our
model enables cinematic storytelling with natural speech and accurate,
context-matched sound effects, producing movie-grade video clips. All the
demos, code and model weights are published at https://aaxwaz.github.io/Ovi
Ovi is a unified audio-video generation model using twin-DiT modules with blockwise cross-modal fusion, enabling natural synchronization and high-quality multimodal outputs.
twin-DiT modules blockwise cross-modal fusion scaled-RoPE embeddings bidirectional cross-attention cinematic storytelling movie-grade video clips