Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual
  Large Language Models
The goal of this work is to enhance balanced multimodal understanding in
audio-visual large language models (AV-LLMs) by addressing modality bias
without requiring additional training. In current AV-LLMs, audio and video
features are typically processed jointly in the decoder. While this strategy
facilitates unified multimodal understanding, it may introduce modality bias,
where the model tends to over-rely on one modality due to imbalanced training
signals. To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet
effective inference-time strategy that requires no additional training or
architectural modifications. FMD first performs modality-specific reasoning by
processing audio-only and video-only inputs through the early decoder layers (a
fork phase), and then merges the resulting hidden states for joint reasoning in
the remaining layers (a merge phase). This approach promotes balanced modality
contributions and leverages complementary information across modalities. We
evaluate our method on two representative AV-LLMs, VideoLLaMA2 and
video-SALMONN, using three benchmark datasets. Experimental results demonstrate
consistent performance improvements on tasks focused on audio, video, and
combined audio-visual reasoning, demonstrating the effectiveness of
inference-time interventions for robust multimodal understanding.
68395a548ead63ba096bba45
Chaeyoung Jung
claimed_verified
2025-06-02T07:46:20.597Z
6770efb5b673f241332fc4a7
/avatars/7aca599492233e6a51c2d6a8f52c644e.svg
Chaeyoung Jung
user
Chae0
68395a548ead63ba096bba46
Youngjoon Jang
68395a548ead63ba096bba47
Jongmin Choi
68395a548ead63ba096bba48
Joon Son Chung
Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual
  Large Language Models
The goal of this work is to enhance balanced multimodal understanding in
audio-visual large language models (AV-LLMs) by addressing modality bias
without requiring additional training. In current AV-LLMs, audio and video
features are typically processed jointly in the decoder. While this strategy
facilitates unified multimodal understanding, it may introduce modality bias,
where the model tends to over-rely on one modality due to imbalanced training
signals. To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet
effective inference-time strategy that requires no additional training or
architectural modifications. FMD first performs modality-specific reasoning by
processing audio-only and video-only inputs through the early decoder layers (a
fork phase), and then merges the resulting hidden states for joint reasoning in
the remaining layers (a merge phase). This approach promotes balanced modality
contributions and leverages complementary information across modalities. We
evaluate our method on two representative AV-LLMs, VideoLLaMA2 and
video-SALMONN, using three benchmark datasets. Experimental results demonstrate
consistent performance improvements on tasks focused on audio, video, and
combined audio-visual reasoning, demonstrating the effectiveness of
inference-time interventions for robust multimodal understanding.
The Fork-Merge Decoding strategy improves balanced multimodal understanding in audio-visual large language models by separating and then combining modality-specific reasoning.
fork-merge decoding AU-LLMs modality bias audio-visual large language models VideoLLaMA2 video-SALMONN benchmark datasets audio-visual reasoning