VMAS: Video-to-Music Generation via Semantic Alignment in Web Music
  Videos
We present a framework for learning to generate background music from video
inputs. Unlike existing works that rely on symbolic musical annotations, which
are limited in quantity and diversity, our method leverages large-scale web
videos accompanied by background music. This enables our model to learn to
generate realistic and diverse music. To accomplish this goal, we develop a
generative video-music Transformer with a novel semantic video-music alignment
scheme. Our model uses a joint autoregressive and contrastive learning
objective, which encourages the generation of music aligned with high-level
video content. We also introduce a novel video-beat alignment scheme to match
the generated music beats with the low-level motions in the video. Lastly, to
capture fine-grained visual cues in a video needed for realistic background
music generation, we introduce a new temporal video encoder architecture,
allowing us to efficiently process videos consisting of many densely sampled
frames. We train our framework on our newly curated DISCO-MV dataset,
consisting of 2.2M video-music samples, which is orders of magnitude larger
than any prior datasets used for video music generation. Our method outperforms
existing approaches on the DISCO-MV and MusicCaps datasets according to various
music generation evaluation metrics, including human evaluation. Results are
available at https://genjib.github.io/project_page/VMAs/index.html
66e25bcb5491a1f2498e5ffe
Yan-Bo Lin
claimed_verified
2024-10-30T09:55:57.782Z
63a715c38729ce5b5640b80b
/avatars/5dc0a167c5dae7756a77d79cfd6334b2.svg
yb
user
genjib
66e25bcb5491a1f2498e5fff
Yu Tian
66e25bcb5491a1f2498e6000
Linjie Yang
66e25bcb5491a1f2498e6001
Gedas Bertasius
admin_assigned
2024-09-12T08:14:06.614Z
65d60993c2c6418adc99206a
/avatars/fb1f9d5af7022bddf4cb090cd477108d.svg
Gedas Bertasius
user
gberta
66e25bcb5491a1f2498e6002
Heng Wang
VMAS: Video-to-Music Generation via Semantic Alignment in Web Music
  Videos
We present a framework for learning to generate background music from video
inputs. Unlike existing works that rely on symbolic musical annotations, which
are limited in quantity and diversity, our method leverages large-scale web
videos accompanied by background music. This enables our model to learn to
generate realistic and diverse music. To accomplish this goal, we develop a
generative video-music Transformer with a novel semantic video-music alignment
scheme. Our model uses a joint autoregressive and contrastive learning
objective, which encourages the generation of music aligned with high-level
video content. We also introduce a novel video-beat alignment scheme to match
the generated music beats with the low-level motions in the video. Lastly, to
capture fine-grained visual cues in a video needed for realistic background
music generation, we introduce a new temporal video encoder architecture,
allowing us to efficiently process videos consisting of many densely sampled
frames. We train our framework on our newly curated DISCO-MV dataset,
consisting of 2.2M video-music samples, which is orders of magnitude larger
than any prior datasets used for video music generation. Our method outperforms
existing approaches on the DISCO-MV and MusicCaps datasets according to various
music generation evaluation metrics, including human evaluation. Results are
available at https://genjib.github.io/project_page/VMAs/index.html
A generative video-music Transformer framework uses semantic and beat alignment to generate diverse and realistic background music from video inputs leveraging a large-scale dataset.
generative video-music Transformer semantic video-music alignment joint autoregressive and contrastive learning video-beat alignment temporal video encoder