FoleyGen: Visually-Guided Audio Generation
Recent advancements in audio generation have been spurred by the evolution of
large-scale deep learning models and expansive datasets. However, the task of
video-to-audio (V2A) generation continues to be a challenge, principally
because of the intricate relationship between the high-dimensional visual and
auditory data, and the challenges associated with temporal synchronization. In
this study, we introduce FoleyGen, an open-domain V2A generation system built
on a language modeling paradigm. FoleyGen leverages an off-the-shelf neural
audio codec for bidirectional conversion between waveforms and discrete tokens.
The generation of audio tokens is facilitated by a single Transformer model,
which is conditioned on visual features extracted from a visual encoder. A
prevalent problem in V2A generation is the misalignment of generated audio with
the visible actions in the video. To address this, we explore three novel
visual attention mechanisms. We further undertake an exhaustive evaluation of
multiple visual encoders, each pretrained on either single-modal or multi-modal
tasks. The experimental results on VGGSound dataset show that our proposed
FoleyGen outperforms previous systems across all objective metrics and human
evaluations.
650a5260e0850b3ff055a141
Xinhao Mei
admin_assigned
2023-09-20T12:07:02.821Z
626c5f8af451470f862534c6
https://cdn-avatars.huggingface.co/v1/production/uploads/626c5f8af451470f862534c6/qL5ubYstO0Won5VOFHviF.jpeg
Xinhao Mei
user
Xinhao
650a5260e0850b3ff055a142
Varun Nagaraja
admin_assigned
2023-09-20T12:07:39.205Z
6435e32a2d0ed796668f9700
/avatars/ef85b2f33d85433eb88e033c302fc68a.svg
Varun Nagaraja
user
varun-nagaraja
650a5260e0850b3ff055a143
Gael Le Lan
650a5260e0850b3ff055a144
Zhaoheng Ni
650a5260e0850b3ff055a145
Ernie Chang
650a5260e0850b3ff055a146
Yangyang Shi
650a5260e0850b3ff055a147
Vikas Chandra
FoleyGen: Visually-Guided Audio Generation
Recent advancements in audio generation have been spurred by the evolution of
large-scale deep learning models and expansive datasets. However, the task of
video-to-audio (V2A) generation continues to be a challenge, principally
because of the intricate relationship between the high-dimensional visual and
auditory data, and the challenges associated with temporal synchronization. In
this study, we introduce FoleyGen, an open-domain V2A generation system built
on a language modeling paradigm. FoleyGen leverages an off-the-shelf neural
audio codec for bidirectional conversion between waveforms and discrete tokens.
The generation of audio tokens is facilitated by a single Transformer model,
which is conditioned on visual features extracted from a visual encoder. A
prevalent problem in V2A generation is the misalignment of generated audio with
the visible actions in the video. To address this, we explore three novel
visual attention mechanisms. We further undertake an exhaustive evaluation of
multiple visual encoders, each pretrained on either single-modal or multi-modal
tasks. The experimental results on VGGSound dataset show that our proposed
FoleyGen outperforms previous systems across all objective metrics and human
evaluations.
FoleyGen, a V2A system employing a Transformer model with novel visual attention mechanisms, achieves superior performance by leveraging bidirectional neural audio codecs and various pretrained visual encoders.
neural audio codec Transformer model visual attention mechanisms visual encoder VGGSound dataset