MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic
  Synchronization
Generating music that aligns with the visual content of a video has been a
challenging task, as it requires a deep understanding of visual semantics and
involves generating music whose melody, rhythm, and dynamics harmonize with the
visual narratives. This paper presents MuVi, a novel framework that effectively
addresses these challenges to enhance the cohesion and immersive experience of
audio-visual content. MuVi analyzes video content through a specially designed
visual adaptor to extract contextually and temporally relevant features. These
features are used to generate music that not only matches the video's mood and
theme but also its rhythm and pacing. We also introduce a contrastive
music-visual pre-training scheme to ensure synchronization, based on the
periodicity nature of music phrases. In addition, we demonstrate that our
flow-matching-based music generator has in-context learning ability, allowing
us to control the style and genre of the generated music. Experimental results
show that MuVi demonstrates superior performance in both audio quality and
temporal synchronization. The generated music video samples are available at
https://muvi-v2m.github.io.
6712157b48e62600e97219d8
Ruiqi Li
6712157b48e62600e97219d9
Siqi Zheng
6712157b48e62600e97219da
Xize Cheng
admin_assigned
2024-10-18T13:38:50.836Z
657543227068a85089680147
/avatars/a0e3371cc018deec7f18a12181eda3b0.svg
xize cheng
user
Exgc
6712157b48e62600e97219db
Ziang Zhang
6712157b48e62600e97219dc
Shengpeng Ji
6712157b48e62600e97219dd
Zhou Zhao
MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic
  Synchronization
Generating music that aligns with the visual content of a video has been a
challenging task, as it requires a deep understanding of visual semantics and
involves generating music whose melody, rhythm, and dynamics harmonize with the
visual narratives. This paper presents MuVi, a novel framework that effectively
addresses these challenges to enhance the cohesion and immersive experience of
audio-visual content. MuVi analyzes video content through a specially designed
visual adaptor to extract contextually and temporally relevant features. These
features are used to generate music that not only matches the video's mood and
theme but also its rhythm and pacing. We also introduce a contrastive
music-visual pre-training scheme to ensure synchronization, based on the
periodicity nature of music phrases. In addition, we demonstrate that our
flow-matching-based music generator has in-context learning ability, allowing
us to control the style and genre of the generated music. Experimental results
show that MuVi demonstrates superior performance in both audio quality and
temporal synchronization. The generated music video samples are available at
https://muvi-v2m.github.io.
MuVi, a novel framework, enhances audio-visual cohesion by generating music that matches video content's mood, theme, rhythm, and pacing using a visual adaptor and a flow-matching-based music generator.
visual adaptor contrastive music-visual pre-training flow-matching-based music generator in-context learning audio-visual synchronization