Masked Generative Video-to-Audio Transformers with Enhanced
  Synchronicity
Video-to-audio (V2A) generation leverages visual-only video features to
render plausible sounds that match the scene. Importantly, the generated sound
onsets should match the visual actions that are aligned with them, otherwise
unnatural synchronization artifacts arise. Recent works have explored the
progression of conditioning sound generators on still images and then video
features, focusing on quality and semantic matching while ignoring
synchronization, or by sacrificing some amount of quality to focus on improving
synchronization only. In this work, we propose a V2A generative model, named
MaskVAT, that interconnects a full-band high-quality general audio codec with a
sequence-to-sequence masked generative model. This combination allows modeling
both high audio quality, semantic matching, and temporal synchronicity at the
same time. Our results show that, by combining a high-quality codec with the
proper pre-trained audio-visual features and a sequence-to-sequence parallel
structure, we are able to yield highly synchronized results on one hand, whilst
being competitive with the state of the art of non-codec generative audio
models. Sample videos and generated audios are available at
https://maskvat.github.io .
6695ec67659a52fa274d8939
Santiago Pascual
6695ec67659a52fa274d893a
Chunghsin Yeh
6695ec67659a52fa274d893b
Ioannis Tsiamas
claimed_verified
2024-07-18T09:08:01.245Z
667572848c58c5a12a18692d
https://cdn-avatars.huggingface.co/v1/production/uploads/667572848c58c5a12a18692d/V-mtLKoOLdIedcSTYZCjn.jpeg
Yiannis  (Ioannis) Tsiamas
user
johntsi
6695ec67659a52fa274d893c
Joan Serrà
Masked Generative Video-to-Audio Transformers with Enhanced
  Synchronicity
Video-to-audio (V2A) generation leverages visual-only video features to
render plausible sounds that match the scene. Importantly, the generated sound
onsets should match the visual actions that are aligned with them, otherwise
unnatural synchronization artifacts arise. Recent works have explored the
progression of conditioning sound generators on still images and then video
features, focusing on quality and semantic matching while ignoring
synchronization, or by sacrificing some amount of quality to focus on improving
synchronization only. In this work, we propose a V2A generative model, named
MaskVAT, that interconnects a full-band high-quality general audio codec with a
sequence-to-sequence masked generative model. This combination allows modeling
both high audio quality, semantic matching, and temporal synchronicity at the
same time. Our results show that, by combining a high-quality codec with the
proper pre-trained audio-visual features and a sequence-to-sequence parallel
structure, we are able to yield highly synchronized results on one hand, whilst
being competitive with the state of the art of non-codec generative audio
models. Sample videos and generated audios are available at
https://maskvat.github.io .
MaskVAT implements a high-quality codec with a sequence-to-sequence masked generative model to produce synchronized and semantically accurate audio from visual video inputs.
video-to-audio generation full-band high-quality audio codec sequence-to-sequence masked generative model temporal synchronicity pre-trained audio-visual features