Video-Guided Foley Sound Generation with Multimodal Controls
Generating sound effects for videos often requires creating artistic sound
effects that diverge significantly from real-life sources and flexible control
in the sound design. To address this problem, we introduce MultiFoley, a model
designed for video-guided sound generation that supports multimodal
conditioning through text, audio, and video. Given a silent video and a text
prompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheels
spinning without wind noise) or more whimsical sounds (e.g., making a lion's
roar sound like a cat's meow). MultiFoley also allows users to choose reference
audio from sound effects (SFX) libraries or partial videos for conditioning. A
key novelty of our model lies in its joint training on both internet video
datasets with low-quality audio and professional SFX recordings, enabling
high-quality, full-bandwidth (48kHz) audio generation. Through automated
evaluations and human studies, we demonstrate that MultiFoley successfully
generates synchronized high-quality sounds across varied conditional inputs and
outperforms existing methods. Please see our project page for video results:
https://ificl.github.io/MultiFoley/
6748776100cf30c297e7d286
Ziyang Chen
claimed_verified
2024-11-28T16:05:44.842Z
6542bf99899592bf11847315
https://cdn-avatars.huggingface.co/v1/production/uploads/6542bf99899592bf11847315/ll0tyrWbbymrhYKOZxRHX.jpeg
Ziyang Chen
user
czyang
6748776100cf30c297e7d287
Prem Seetharaman
6748776100cf30c297e7d288
Bryan Russell
6748776100cf30c297e7d289
Oriol Nieto
6748776100cf30c297e7d28a
David Bourgin
6748776100cf30c297e7d28b
Andrew Owens
6748776100cf30c297e7d28c
Justin Salamon
Video-Guided Foley Sound Generation with Multimodal Controls
Generating sound effects for videos often requires creating artistic sound
effects that diverge significantly from real-life sources and flexible control
in the sound design. To address this problem, we introduce MultiFoley, a model
designed for video-guided sound generation that supports multimodal
conditioning through text, audio, and video. Given a silent video and a text
prompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheels
spinning without wind noise) or more whimsical sounds (e.g., making a lion's
roar sound like a cat's meow). MultiFoley also allows users to choose reference
audio from sound effects (SFX) libraries or partial videos for conditioning. A
key novelty of our model lies in its joint training on both internet video
datasets with low-quality audio and professional SFX recordings, enabling
high-quality, full-bandwidth (48kHz) audio generation. Through automated
evaluations and human studies, we demonstrate that MultiFoley successfully
generates synchronized high-quality sounds across varied conditional inputs and
outperforms existing methods. Please see our project page for video results:
https://ificl.github.io/MultiFoley/
MultiFoley generates high-quality synchronized sounds for videos using text, audio, and video inputs, outperforming existing methods.
MultFoley multimodal conditioning video-guided sound generation internet video datasets professional SFX recordings full-bandwidth audio generation