UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
We introduce UniVerse-1, a unified, Veo-3-like model capable of
simultaneously generating coordinated audio and video. To enhance training
efficiency, we bypass training from scratch and instead employ a stitching of
experts (SoE) technique. This approach deeply fuses the corresponding blocks of
pre-trained video and music generation experts models, thereby fully leveraging
their foundational capabilities. To ensure accurate annotations and temporal
alignment for both ambient sounds and speech with video content, we developed
an online annotation pipeline that processes the required training data and
generates labels during training process. This strategy circumvents the
performance degradation often caused by misalignment text-based annotations.
Through the synergy of these techniques, our model, after being finetuned on
approximately 7,600 hours of audio-video data, produces results with
well-coordinated audio-visuals for ambient sounds generation and strong
alignment for speech generation. To systematically evaluate our proposed
method, we introduce Verse-Bench, a new benchmark dataset. In an effort to
advance research in audio-video generation and to close the performance gap
with state-of-the-art models such as Veo3, we make our model and code publicly
available. We hope this contribution will benefit the broader research
community. Project page: https://dorniwang.github.io/UniVerse-1/.
68bf827d207285de11b07b2d
Duomin Wang
claimed_verified
2025-09-09T13:48:24.856Z
64ae9b88a22a179fc4d07992
/avatars/c9065f04a1188ea3129e56a90328ffd3.svg
wang
user
dorni
68bf827d207285de11b07b2e
Wei Zuo
68bf827d207285de11b07b2f
Aojie Li
68bf827d207285de11b07b30
Ling-Hao Chen
68bf827d207285de11b07b31
Xinyao Liao
68bf827d207285de11b07b32
Deyu Zhou
68bf827d207285de11b07b33
Zixin Yin
68bf827d207285de11b07b34
Xili Dai
68bf827d207285de11b07b35
Daxin Jiang
68bf827d207285de11b07b36
Gang Yu
UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
We introduce UniVerse-1, a unified, Veo-3-like model capable of
simultaneously generating coordinated audio and video. To enhance training
efficiency, we bypass training from scratch and instead employ a stitching of
experts (SoE) technique. This approach deeply fuses the corresponding blocks of
pre-trained video and music generation experts models, thereby fully leveraging
their foundational capabilities. To ensure accurate annotations and temporal
alignment for both ambient sounds and speech with video content, we developed
an online annotation pipeline that processes the required training data and
generates labels during training process. This strategy circumvents the
performance degradation often caused by misalignment text-based annotations.
Through the synergy of these techniques, our model, after being finetuned on
approximately 7,600 hours of audio-video data, produces results with
well-coordinated audio-visuals for ambient sounds generation and strong
alignment for speech generation. To systematically evaluate our proposed
method, we introduce Verse-Bench, a new benchmark dataset. In an effort to
advance research in audio-video generation and to close the performance gap
with state-of-the-art models such as Veo3, we make our model and code publicly
available. We hope this contribution will benefit the broader research
community. Project page: https://dorniwang.github.io/UniVerse-1/.
UniVerse-1, a unified audio-video generation model, uses a stitching of experts technique to combine pre-trained video and music models, ensuring accurate temporal alignment and producing high-quality audio-visual outputs.
unified model Veo-3-like model stitching of experts (SoE) pre-trained video generation pre-trained music generation online annotation pipeline temporal alignment ambient sounds speech generation Verse-Bench audio-video generation