Taming Multimodal Joint Training for High-Quality Video-to-Audio
  Synthesis
We propose to synthesize high-quality and synchronized audio, given video and
optional text conditions, using a novel multimodal joint training framework
MMAudio. In contrast to single-modality training conditioned on (limited) video
data only, MMAudio is jointly trained with larger-scale, readily available
text-audio data to learn to generate semantically aligned high-quality audio
samples. Additionally, we improve audio-visual synchrony with a conditional
synchronization module that aligns video conditions with audio latents at the
frame level. Trained with a flow matching objective, MMAudio achieves new
video-to-audio state-of-the-art among public models in terms of audio quality,
semantic alignment, and audio-visual synchronization, while having a low
inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio
also achieves surprisingly competitive performance in text-to-audio generation,
showing that joint training does not hinder single-modality performance. Code
and demo are available at: https://hkchengrex.github.io/MMAudio
6768d18826eb881162077014
Ho Kei Cheng
claimed_verified
2024-12-23T11:09:58.995Z
63041b541dd5d3c62486c294
/avatars/a5286d562f7b9082730f760e66c3bf29.svg
Ho Kei Cheng
user
hkchengrex
6768d18826eb881162077015
Masato Ishii
admin_assigned
2024-12-23T11:33:45.656Z
674545617e0ea169b0c471d1
/avatars/a422e3efa5fb1c3f2c6c0997c412b088.svg
Masato Ishii
user
mi141
6768d18826eb881162077016
Akio Hayakawa
claimed_verified
2024-12-30T19:36:17.816Z
66d54244b4396d43c356ea2e
/avatars/c90dde293cbda625db73b35d8f191328.svg
Akio Hayakawa
user
AkHyk
6768d18826eb881162077017
Takashi Shibuya
claimed_verified
2024-12-23T11:09:56.433Z
6650773ca6acfdd2aba7d486
/avatars/d297886ea60dbff98a043caf825820ed.svg
Takashi Shibuya
user
TakashiShibuyaSony
6768d18826eb881162077018
Alexander Schwing
6768d18826eb881162077019
Yuki Mitsufuji
admin_assigned
2024-12-23T11:34:10.186Z
665e32384ecc8a7181634f6d
/avatars/8752f952010540d14f45eac849e91371.svg
Yuki Mitsufuji
user
mittu1204
Taming Multimodal Joint Training for High-Quality Video-to-Audio
  Synthesis
We propose to synthesize high-quality and synchronized audio, given video and
optional text conditions, using a novel multimodal joint training framework
MMAudio. In contrast to single-modality training conditioned on (limited) video
data only, MMAudio is jointly trained with larger-scale, readily available
text-audio data to learn to generate semantically aligned high-quality audio
samples. Additionally, we improve audio-visual synchrony with a conditional
synchronization module that aligns video conditions with audio latents at the
frame level. Trained with a flow matching objective, MMAudio achieves new
video-to-audio state-of-the-art among public models in terms of audio quality,
semantic alignment, and audio-visual synchronization, while having a low
inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio
also achieves surprisingly competitive performance in text-to-audio generation,
showing that joint training does not hinder single-modality performance. Code
and demo are available at: https://hkchengrex.github.io/MMAudio
A multimodal framework jointly trains with video and text to generate high-quality, semantically aligned, and visually synchronized audio, achieving state-of-the-art performance.
multimodal joint training MMAudio text-audio data semantic alignment conditional synchronization module audio latents frame level flow matching objective audio quality audio-visual synchronization text-to-audio generation low inference time