AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal
  Audio-Video Generation
We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video
generation that leverages the activations of frozen video and audio diffusion
models for temporally-aligned cross-modal conditioning. The key to our
framework is a Fusion Block that enables bidirectional information exchange
between our backbone video and audio diffusion models through a
temporally-aligned self attention operation. Unlike prior work that uses
feature extractors pretrained for other tasks for the conditioning signal,
AV-Link can directly leverage features obtained by the complementary modality
in a single framework i.e. video features to generate audio, or audio features
to generate video. We extensively evaluate our design choices and demonstrate
the ability of our method to achieve synchronized and high-quality audiovisual
content, showcasing its potential for applications in immersive media
generation. Project Page: snap-research.github.io/AVLink/
6765d9f5bde4bc579f5b0603
Moayed Haji-Ali
claimed_verified
2025-06-25T08:10:50.582Z
665aa3c1957e1e79ff031acb
/avatars/8fbe062bc8c635337991b5bf90c07052.svg
Moayed Haji Ali
user
mali6
6765d9f5bde4bc579f5b0604
Willi Menapace
6765d9f5bde4bc579f5b0605
Aliaksandr Siarohin
6765d9f5bde4bc579f5b0606
Ivan Skorokhodov
6765d9f5bde4bc579f5b0607
Alper Canberk
6765d9f5bde4bc579f5b0608
Kwot Sin Lee
6765d9f5bde4bc579f5b0609
Vicente Ordonez
6765d9f5bde4bc579f5b060a
Sergey Tulyakov
AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal
  Audio-Video Generation
We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video
generation that leverages the activations of frozen video and audio diffusion
models for temporally-aligned cross-modal conditioning. The key to our
framework is a Fusion Block that enables bidirectional information exchange
between our backbone video and audio diffusion models through a
temporally-aligned self attention operation. Unlike prior work that uses
feature extractors pretrained for other tasks for the conditioning signal,
AV-Link can directly leverage features obtained by the complementary modality
in a single framework i.e. video features to generate audio, or audio features
to generate video. We extensively evaluate our design choices and demonstrate
the ability of our method to achieve synchronized and high-quality audiovisual
content, showcasing its potential for applications in immersive media
generation. Project Page: snap-research.github.io/AVLink/
AV-Link is a unified framework using frozen diffusion models and a Fusion Block with self-attention for generating temporally aligned audio-visual content.
diffusion models Fusion Block self attention cross-modal conditioning audio-to-video generation video-to-audio generation immersive media generation