Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
  Condition For Foley Sound
Foley sound synthesis is crucial for multimedia production, enhancing user
experience by synchronizing audio and video both temporally and semantically.
Recent studies on automating this labor-intensive process through
video-to-sound generation face significant challenges. Systems lacking explicit
temporal features suffer from poor controllability and alignment, while
timestamp-based models require costly and subjective human annotation. We
propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a
temporal event condition with semantic timbre prompts (audio or text). RMS, a
frame-level intensity envelope feature closely related to audio semantics,
ensures high controllability and synchronization. The annotation-free
self-supervised learning framework consists of two stages, Video2RMS and
RMS2Sound, incorporating novel ideas including RMS discretization and
RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation
shows that Video-Foley achieves state-of-the-art performance in audio-visual
alignment and controllability for sound timing, intensity, timbre, and nuance.
Code, model weights, and demonstrations are available on the accompanying
website. (https://jnwnlee.github.io/video-foley-demo)
66c7f43e9a13dc8fc2310324
Junwon Lee
claimed_verified
2024-08-26T07:15:43.942Z
6555d537f47c458e89e926cc
/avatars/3df074b6257473e7c98fb9d97fac9196.svg
Junwon Lee
user
jnwnlee
66c7f43e9a13dc8fc2310325
Jaekwon Im
66c7f43e9a13dc8fc2310326
Dabin Kim
66c7f43e9a13dc8fc2310327
Juhan Nam
Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
  Condition For Foley Sound
Foley sound synthesis is crucial for multimedia production, enhancing user
experience by synchronizing audio and video both temporally and semantically.
Recent studies on automating this labor-intensive process through
video-to-sound generation face significant challenges. Systems lacking explicit
temporal features suffer from poor controllability and alignment, while
timestamp-based models require costly and subjective human annotation. We
propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a
temporal event condition with semantic timbre prompts (audio or text). RMS, a
frame-level intensity envelope feature closely related to audio semantics,
ensures high controllability and synchronization. The annotation-free
self-supervised learning framework consists of two stages, Video2RMS and
RMS2Sound, incorporating novel ideas including RMS discretization and
RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation
shows that Video-Foley achieves state-of-the-art performance in audio-visual
alignment and controllability for sound timing, intensity, timbre, and nuance.
Code, model weights, and demonstrations are available on the accompanying
website. (https://jnwnlee.github.io/video-foley-demo)
Video-Foley achieves top performance in video-to-sound generation by using RMS for temporal control and semantic timbre prompts without human annotation.
RMS Root Mean Square intensity envelope temporal event condition semantic timbre prompts annotation-free self-supervised learning Video2RMS RMS2Sound RMS discretization RMS-ControlNet text-to-audio model audio-visual alignment