^RFLAV: Rolling Flow matching for infinite Audio Video generation
Joint audio-video (AV) generation is still a significant challenge in
generative AI, primarily due to three critical requirements: quality of the
generated samples, seamless multimodal synchronization and temporal coherence,
with audio tracks that match the visual data and vice versa, and limitless
video duration. In this paper, we present , a novel transformer-based
architecture that addresses all the key challenges of AV generation. We explore
three distinct cross modality interaction modules, with our lightweight
temporal fusion module emerging as the most effective and computationally
efficient approach for aligning audio and visual modalities. Our experimental
results demonstrate that  outperforms existing state-of-the-art models
in multimodal AV generation tasks. Our code and checkpoints are available at
https://github.com/ErgastiAlex/R-FLAV.
67d140378cb4592900a1a75e
Alex Ergasti
claimed_verified
2025-03-12T08:36:03.978Z
66895b3d41fcf83c026b5dca
/avatars/f1104041ee3445024f05d5d0b4d1550b.svg
Alex Ergasti
user
MaverickAlex
67d140378cb4592900a1a75f
Giuseppe Gabriele Tarollo
67d140378cb4592900a1a760
Filippo Botti
67d140378cb4592900a1a761
Tomaso Fontanini
67d140378cb4592900a1a762
Claudio Ferrari
67d140378cb4592900a1a763
Massimo Bertozzi
67d140378cb4592900a1a764
Andrea Prati
^RFLAV: Rolling Flow matching for infinite Audio Video generation
Joint audio-video (AV) generation is still a significant challenge in
generative AI, primarily due to three critical requirements: quality of the
generated samples, seamless multimodal synchronization and temporal coherence,
with audio tracks that match the visual data and vice versa, and limitless
video duration. In this paper, we present , a novel transformer-based
architecture that addresses all the key challenges of AV generation. We explore
three distinct cross modality interaction modules, with our lightweight
temporal fusion module emerging as the most effective and computationally
efficient approach for aligning audio and visual modalities. Our experimental
results demonstrate that  outperforms existing state-of-the-art models
in multimodal AV generation tasks. Our code and checkpoints are available at
https://github.com/ErgastiAlex/R-FLAV.
A novel transformer-based architecture addresses challenges in audio-video generation by effectively aligning audio and visual modalities using a lightweight temporal fusion module.
transformer-based architecture cross modality interaction modules temporal fusion module multimodal AV generation