Diffusion-Link: Diffusion Probabilistic Model for Bridging the
  Audio-Text Modality Gap
Contrastive audio-language pretraining yields powerful joint representations,
yet a persistent audio-text modality gap limits the benefits of coupling
multimodal encoders with large language models (LLMs). We present
Diffusion-Link, a diffusion-based modality-bridging module that generatively
maps audio embeddings into the text-embedding distribution. The module is
trained at the output embedding from the frozen multimodal encoder and
implemented as a lightweight network with three residual MLP blocks. To assess
the effect of Diffusion-Link on multimodal encoder-LLM coupling, we evaluate on
Automatic Audio Captioning (AAC); to our knowledge, this is the first
application of diffusion-based modality bridging to AAC. We report two results.
(1) Modality-gap analysis: on similarity and geometric criteria, Diffusion-Link
reduces the modality gap the most among prior diffusion-based methods and shows
a collective migration of audio embeddings toward the text distribution. (2)
Downstream AAC: attaching Diffusion-Link to the same multimodal LLM baseline
achieves state-of-the-art on AudioCaps in both zero-shot and fully supervised
captioning without external knowledge, with relative gains up to 52.5% and
7.5%, respectively. These findings show that closing the modality gap is
pivotal for effective coupling between multimodal encoders and LLMs, and
diffusion-based modality bridging offers a promising direction beyond
knowledge-retrieval-centric designs. Code will be released upon acceptance
https://github.com/DevKiHyun/Diffusion-Link
68ef61fa121c484bb15d2db4
KiHyun Nam
claimed_verified
2025-10-15T15:24:26.040Z
67061a90d66c2d3af1956070
/avatars/3f9ea921ce621363e0f2ac1f03ba99da.svg
Kihyun Nam
user
southKH
68ef61fa121c484bb15d2db5
Jongmin Choi
68ef61fa121c484bb15d2db6
Hyeongkeun Lee
68ef61fa121c484bb15d2db7
Jungwoo Heo
68ef61fa121c484bb15d2db8
Joon Son Chung
Diffusion-Link: Diffusion Probabilistic Model for Bridging the
  Audio-Text Modality Gap
Contrastive audio-language pretraining yields powerful joint representations,
yet a persistent audio-text modality gap limits the benefits of coupling
multimodal encoders with large language models (LLMs). We present
Diffusion-Link, a diffusion-based modality-bridging module that generatively
maps audio embeddings into the text-embedding distribution. The module is
trained at the output embedding from the frozen multimodal encoder and
implemented as a lightweight network with three residual MLP blocks. To assess
the effect of Diffusion-Link on multimodal encoder-LLM coupling, we evaluate on
Automatic Audio Captioning (AAC); to our knowledge, this is the first
application of diffusion-based modality bridging to AAC. We report two results.
(1) Modality-gap analysis: on similarity and geometric criteria, Diffusion-Link
reduces the modality gap the most among prior diffusion-based methods and shows
a collective migration of audio embeddings toward the text distribution. (2)
Downstream AAC: attaching Diffusion-Link to the same multimodal LLM baseline
achieves state-of-the-art on AudioCaps in both zero-shot and fully supervised
captioning without external knowledge, with relative gains up to 52.5% and
7.5%, respectively. These findings show that closing the modality gap is
pivotal for effective coupling between multimodal encoders and LLMs, and
diffusion-based modality bridging offers a promising direction beyond
knowledge-retrieval-centric designs. Code will be released upon acceptance
https://github.com/DevKiHyun/Diffusion-Link
Diffusion-Link, a diffusion-based modality-bridging module, reduces the audio-text modality gap and enhances multimodal encoder-LLM coupling, achieving state-of-the-art performance in automatic audio captioning.
contrastive audio-language pretraining diffusion-based modality-bridging audio embeddings text-embedding distribution residual MLP blocks Automatic Audio Captioning modality-gap analysis similarity criteria geometric criteria zero-shot captioning fully supervised captioning AudioCaps